Often, Less is Better!

When I was learning Ruby, I was searching for things that could be improved. I was looking for ways I could make things better. One of the areas I was involved in was testing. This should come as no surprise. If you are writing Ruby code, you need to be writing tests. The exact order in which this is done is not my point. Just that tests must be written and run frequently. Here is a typical test run using the minitest facility:

Standard Test Output

To my mind, this was not communicating that true progress of the tests. Consider: What version of minitest was in use? What files were being processed? How many tests were in each of those files? Most of all, why does this output look so boring?

Yes; boring! So being bored I thought: Surely I can do better! I came up with a gem I call minitest_visible to spruce things up. Here’s what its output looks like:

Enhanced Test Output

Much better! Clearer! Nice progress! and so entertaining? Well at least I thought it was a big improvement. It would seem that not many people shared my view. I can see their points. Notice that the bottom line is the same in both cases:

21 runs, 92 assertions, 0 failures, 0 errors, 0 skips

This result line is what counts. These are tests, not video games. They should test the code without a lot of bells and whistles and let you get on with the real work.

I was also told that my little gem did not fit in with the minitest ecosystem. I asked what was meant by this but I never got a reply. I suppose somebody was channeling their inner Linus Torvalds of rudeness that day. I’ll never know. It also does not matter.

I have finally come to realize that the critics were right. The best answer is to keep things as simple and lean as possible. No being fancy; no showing off. To that end I have worked to remove the minitest_visible augmentation from almost all of my work.

Affected gems are: composite_rng, counted_cache, fibonacci_rng, flex_array, format_engine, full_clone, full_dup, fully_freeze, in_array, insouciant, lexical_analyzer, make_gem, mini_erb, mini_readline, mini_term, mysh, parse_queue, pause_output, safe_clone, and safe_dup. The exception is the fOOrth gem experiment that has so many test files that it would be hard to sort out what was happening without a little help. At least for now.

Now these gems can be used by others without the hassle of having to install a non-standard testing extension.

Is less more? I’m not sure. Is less better? Very often, yes it is.

Your truly

Peter Camilleri (aka Squidly Jones)

Work In Progress

At this time, I would like to write about a project that I have been working on for several months. This project is based on the awesome Commander X 16 project created by YouTuber Dave Murray (The 8-bit Guy). Here are two videos where he goes into his vision for the project:

To be clear, other than being a member of the project’s Facebook group, I have no direct involvement with this project. Still, I thought it might be interesting to explore aspects of this new machine. Some ideas that came to mind were:

  • Documenting interesting aspects of the W65C02S processor, especially how this was different from the classic MOS TECH 6502 of years gone by. Special attention was paid to bug fixes for old problems and those quirks that still remain to frustrate the unprepared.
  • Creating an easy to understand memory map of the I/O page and the vital first four pages of RAM.
  • Writing some utility macros to ease the job of writing applications in assembly language. So far these include 16 bit utilities and enhanced branches.
  • Exploring interesting aspects of the design like memory mapping, bank switching, and timing.
  • Looking at virtual machine design as it could apply to this computer.

This has gone fairly well, but now things have been shifting about a bit. The study of virtual machines, began as a simple port of the old Apple ][ Sweet-16 soft processor. Based on information published in the May and November 1977 issues of Byte magazine, I created a port of this interpreter to the newer chip, using newer development tools.

I just couldn’t leave it there though. I then embarked on a deep dive look at the design of virtual machine interpreters from the point of view of implementation on the Commander X 16 and its W65C02S processor.

To be fair, I know that it’s gotten a bit out of hand. The scope of this effort just keeps growing. Soon, I will have to come to my senses and split the VM study off into a separate project so that the Commander X 16 part can be wrapped up.

For now, they are still together. You can see what has been written, warts and all, here on GitHub. Just remember that this is still very much… a work in progress.

Yours Truly;

Peter Camilleri (aka Squidly Jones)

Updating Dependencies

There are things you don’t want to see when you review your code. Here’s one:

Potential Security Vulnerability Warning Alert

What’s worse, this message was hard to track down. There was no way (that I found) to have GitHub tell me about all the code that was affected. I’d have to check each one manually until I had found and fixed them all.

I had several things going for me. The alerts were all caused by one thing; the discovery of a bug in the “rake” gem. Here’s one version of the bug report I found:

It was discovered that Rake incorrectly handled certain files. An attacker could use this issue to possibly execute arbitrary commands.

USN-4295-1: Rake vulnerability

Now, in theory, one could do an in depth analysis of this bug and conclude that it has no impact on your code. I do not recommend this approach. I would not have confidence that I could see all the devious ways this fault could be exploited. Further, I know I cannot possibly predict every move by the criminals. That is why I will now focus on ensuring that my code specifies a version of rake that does not have this defect. That is we need to specify versions of rake with version greater than or equal to 12.3.3.

I write Ruby Gems and for those gems, dependencies are specified in the <gem_name>.gemspec file. Let’s see some rake dependency entries and see how they stack up to our security requirement:

spec.add_development_dependency “rake”, “~> 10.0”

This was the dependency entry that generated my alert messages. To translate into plain English, it says that any version of rake is OK, so long as it was version 10.something. The last such release was 10.5.0 which clearly has the bug and thus generates a warning. This entry clearly must be replaced.

Now many of my Ruby Gems had the following entry that did not result in a warning, but still leaves something to be desired:

spec.add_development_dependency “rake”, “~> 12.0”

This entry says to accept any version of rake so long as it was version 12.something. The last such release was 12.3.3 which is clear and free of the bug! So, yes, this is much better, but it suffers from the fact that it limits rake to version 12. Version 13 is already out and forcing the use of an older version may not be a good idea. I chose to replace these entries as well.

So, we now examine the simplest and most permissive specification:

spec.add_development_dependency ‘rake’

This translates to anythings goes, but use the very latest if it’s not already installed. This is very simple, it generated no warnings, and it can get the latest. It will also settle for much older versions, including those with the bug. This too must be updated.

This is the entry recommended for use by the alert:

spec.add_development_dependency “rake”, “>= 12.3.3”

This translates to “Anything from 12.3.3 or later”. This ensures that we will not use older versions of (buggy) code, and that we won’t exclude newer versions of code either.

Now there may be legitimate reasons to stick to older code. I know there are working web sites using ancient versions of rails running on equally old versions of ruby. Even so, there are (too many to list here) real risks to running obsolete code. If at all possible, keep up-to-date. It may require some effort, but it is worth it. Test your code with newer versions before pushing them out into the wild. If problems/issues arise, fix them and update your dependencies. This usually can be accomplished in a modest amount of time. If it can’t then you need to lock in the obsolete code dependencies and take a serious look at re-engineering your application.

In the end, everything needs maintenance. The reason we all avoid gems that have gone many years with no updates is that we know that nobody is tending house. I was caught by surprise and it is hard to anticipate what the next disruption will be, but by defensively coding, the risk can be reduced.

Finally let me take a moment to list which of my gems needed to be updated due to this issue. If you have a current, safe version of rake installed, you need do nothing even if you are using these gems: composite_rng, connect_n_game, counted_cache, fibonacci_rng, format_engine, format_output, full_clone, full_dup, fully_freeze, in_array. insouciant, lexical_analyzer, make_gem, mini_erb, mini_readline, mini_term, mysh, parse_queue, pause_output, ruby_sscanf, safe_clone, safe_dup, test65, and vls. In addition, fOOrth, games_lessons, and rctp were updated but the changes are currently part of unreleased code.

Yours Truly

Peter Camilleri (aka Squidly Jones)

A World Upside-Down.

The Man with the Golden Gun; Actual stunt, no CGI existed in 1974!

So, while this a technically oriented blog, it is not possible to exist outside the rest of the world. In particular, the absolute chaos created by the Covid-19 pandemic. Things are not normal. Not at all. Travel is being discouraged, government services of all sorts are being closed. Shows at theaters; classes in schools; poverty in casinos; club meetings and lunches; all canceled for the foreseeable future.

It is not my intent to make another P.S.A. about what needs to be done. Anyone not living under a rock already knows. Nor do I intend to a launch a polemic on the evils of the mass mistreatment of animals that is the root cause of this catastrophe. Nope. Instead, I would like to share a memory of a freer, happier time. A distant time I like to call “Last Month”.

In Ontario, the third Monday of the month of February is designated as the Family Day holiday. On the Saturday of the preceding weekend, we get treated to the Durham DIY Showcase. What is that, you ask?

It’s a chance to look at work of local hobbyists of all sorts; an event with free admission no less. There was amateur (ham) radio, impressive model railroad setups and much more. One item that caught my eye was this amazing, wooden geared, clock:

I must admit I should have taken a lot more photographs at the event. On the bright side, at least I took this one. I suppose that while all the stuff was interesting, the clock was an amazing mix of art, engineering, and functionality.

Also present was the Durham Personal Computer User’s Club (DPCUC) of which I am a member. I really like the retro computers, Arduinos, and Raspberry (and other) PI machines. Again no pics, my bad.

Also present was a little display of gorgeous minerals. Here are the ones I purchased:

Now as I was looking over the stones and making my selections, there was a concerted effort to inform me of the mystical healing “powers” of these stones. I must admit that I sort of cut them off. I did not need anyone telling me about powers I already knew about. Like the power of things of beauty to inspire us and lift our spirits. Joy is good for the soul.

And this ends this little mini-report. This event should return next year on Family Day and hopefully this time I’ll take more pictures. Even better, maybe I will have a table of my own displaying constructions of interest. In the meantime, I think I’ve gotten my head around the mechanics of mixing text and pictures in WordPress.

Now, WASH YOUR HANDS! Did you think I wouldn’t notice? Warm water, soap, 20 seconds, now!

I’m Back! (Again)

I’d be a liar if I didn’t admit that I have let this web site languish for a very long time. Furthermore, this is not my first attempt to raise it from the dead, even though this last quiet spell is the longest one to date.

It’s been such a long pause in fact that a lot of how WordPress works and is used have changed since the last time I wrote a blog entry. I have a lot to learn before I regain a decent level of proficiency in writing these entries. I suppose I should have the confidence to know that I will figure it all, or enough, out, eventually.

So what’s so different? What makes me think that this reboot will be any different than all the previous failures? A lot has changed: In the intervening years, I became ensnared in a “business” that devoured my time and resources and left me with little to show for it besides notebooks full of hollow slogans and ideas; I became the sole caregiver to two disabled seniors, the care of whom drained all my energy, time, and stamina; and I lost all of my immediate family and suffered, no am still suffering through a horribly painful mourning process. That pain, though crippling, is ever so slowly beginning to diminish.

Yet now, scant weeks ahead of the surging Covid-19 pandemic, I enjoyed a desperately needed vacation in Arizona with my friend Bret and his partner Adam, and their friends Joe and Roger. Two weeks without stress or care. Two weeks of vital healing. Perhaps there is light in the universe after all?

So a lot has changed. I find myself with no responsibilities beyond myself for the first time in 10 years. I have the time, and increasingly, the energy to DO something with my life again. I am slowly taking back my life and the feeling that I can affect its trajectory.

The reality remains: Talk is cheap; Content is King. So I currently have a slate of five new articles in the pipeline. I plan to post at least once a week, and more when topics are abundant. Above all, I plan to write and keep writing. This blog will bear testimony to this in the coming weeks.

I very much doubt that I have anybody left of the cadre who followed my blog. I will need to attract new followers all the while I find new and exciting blogs for me to follow. I’m back… and I have the combination! 😉

PS: I have not forgotten Rascal and Buddy, Adam’s wonderful cats! They made my stay even more special.

Shout Out!

Hello everyone!

Lately, I’ve been getting a lot of notification messages about new users joining this blog. I just checked the numbers and WordPress tells me that there are 7,264 of you out there in cyber-space. Wow! Hard to believe, over seven thousand.

Then I think: That’s a huge number for a blog with so little current activity. In last three days, this blog received only 19 visits. In the same period, 58 subscribers were added. How can that be a thing? How is it even possible? What’s up?

So this post is all about you, the readers and subscribers to this blog. That is the real yous out there. This is your chance to show that you are not web hack or a digital hallucination. How? Write a comment! I’d like very much to hear from you. What do you like about the blog? What would you like to see covered? What’s on your mind? Say anything you like within the bounds of good taste. Still, if you think the blog sucks, say so!  😉

The opinions of real people matter!

Best regards;

Peter Camilleri

Please note that comments with promotions, spam, or links will NOT be accepted.

Survey Results: Introduction [Updated]

In April of this year, I completed gathering the raw data for my survey of gem downloads from my postings at the RubyGems web site. In the past, I have published progress reports at the 12 week and 24 week progress points. I am glad that the data collection is finally done. I admit, that it was only about an hour a week, but after several months it began to take on a level of tedium and boredom.

To be clear, data was collected for all 52 weeks, however, the data for week, 41, was lost in system crash. The data for that week is interpolated from the data of weeks 40 and 42.

The $65,536 question is WHY? Data collections and studies are are a lot of work and normally only undertaken with a goal in mind. My goal was simply to understand who was downloading the Ruby code I was writing and giving to the world. Unlike really popular authors, I did not have the luxury of looking to see thousands of downloads a day. My numbers were quite meager. My hypothesis was that by gathering detailed data, over a reasonably long span of time, some truths could be gleaned from that data that would reveal the nature of the user base. That is, assuming such a user base even existed.

You see, my fear is that I am just a crazy lunatic, working away in complete isolation, writing code that nobody will ever read. Just thinking about this scenario makes breathing difficult and causes my gut to twist and contort painfully.

The data itself reveals a brighter side. While subtle, there are some signs that there real are people out there at least looking at this code, and maybe incorporating it into their own projects.

There is a lot of data, too much for this blog or any single posting. In the coming days, I plan a number of articles looking at the data and coming to some concrete conclusions about Ruby Gem downloads and how to gage the success of a code release and maybe some tips for success in the world of software components.

Best regards

Peter Camilleri

ps: My library of gems can be found at Ruby Gems and the source code is at GitHub.

Another digit please!

Well, it’s a smallish milestone, but I’ve finally made it to the six digit club! Here take a look at this ego gratifying selfie:

I must say that my download study has been given a boost last week as weekly downloads hit an all time high of 3,335. I suspect this may have something to do with my increased level of activity on the stackoverflow web site, but that is largely speculation, It could just as easily be one of those volume blips that occur from time-to-time. I know however that it is not driven be release activity as only one gem was updated that week.

Well onward and upward.

Best regards

Peter Camilleri

 

Unconventional tools, #2

Hi All;

Now, we’ve all seen code that at least appears to work, but is very poorly written. You might even say the code “stinks”. Now, it can be really hard to detect code smells in your own code. A major air freshener company speaks of people going “noseblind” to smells. The old solution to this problem was to have code reviews. This never works. People gloss over problems, the reviews take forever, and feelings can be hurt or axes can be ground. What is needed is an objective, easy-to-perform, review of the code. What is needed is an automated tool!

For me, programming in free ruby, the tool is called… reek!

Now you may think: “Why do I need a tool to complain about my working code?” I must admit, I sometimes think that too. The answer to that question is that whenever I dig into the code and clean up the smells, the result is almost always much better code. Now, I know that’s a serious claim, so I am going to back it up with a real example from my own code. The following bit of code is used to process snippets of ruby code embedded in strings and surrounded by {{  }} structures. They generally go by the name “handlebars”. Here is our starting point:

#Process a string with code embedded in handlebars and 
#backslash quotes.
def eval_handlebars(in_str)
  out_str = ""

  loop do
    pre_match, match, in_str = in_str.partition(/{{.*?}}/m)
    out_str << pre_match
    return out_str.gsub(/\\\S/) {|found| found[1]} if match.empty?

    code = match[2...-2]
    silent = code.end_with?("#")
    result = instance_eval(code)
    out_str << result.to_s unless silent
  end
end

 When I ran reek against this code, it made the following observations:

lib/mysh/user_input/handlebars.rb -- 2 warnings:
 [25, 27, 32]:FeatureEnvy: Object#eval_handlebars refers to
  out_str more than self (maybe move it to another class?)
 [19]:TooManyStatements: Object#eval_handlebars has approx 10 
  statements
2 total warnings

Well, the code is pretty ugly, so perhaps this is not surprising. So; how to proceed? The first clue comes from the comment line.

#Process a string with code embedded in handlebars and 
#backslash quotes.

This method is doing two distinct things! Maybe it should be two distinct methods? So here’s the first fixup:

 
#Process a string with code embedded in handlebars and 
#backslash quotes.
def eval_handlebars(str)
  do_process_handlebars(str).gsub(/\\\S/) {|found| found[1]}
end

private

#Process a string with code embedded in handlebars.
def do_process_handlebars(in_str)
  out_str = ""

  loop do
    pre_match, match, in_str = in_str.partition(/{{.*?}}/m)
    out_str << pre_match
    return out_str if match.empty?

    code = match[2...-2]
    silent = code.end_with?("#")
    result = instance_eval(code)
    out_str << result.to_s unless silent
  end
end

The new method do_process_handlebars just does handlebars while the original method takes that result and processes backslash quotes. So what does reek think of the code now?

lib/mysh/user_input/handlebars.rb -- 1 warning:
 [26]:TooManyStatements: Object#do_process_handlebars has approx 9 
  statements
1 total warning

This is a lot better, but now my new method is too long! What to do? I could ignore the problem; I could tell reek to just ignore the problem; or I could mashup the code and use sneaky tricks to make it use fewer lines. None of these are good choices here. Instead, let’s really look at what the code is doing. We see a loop, processing a string by repeatedly looking for a regular expression and then performing a substitution of the found text with new text. Wow! Did I really write that code? What we have here is a kludgy reinvention of the gsub method! The very same method already used to perform the backslash quoting. Zounds!

OK; so let’s see what happens when we replace the kludge of gsub with that actual gsub method:

#Process a string with code embedded in handlebars.
def do_process_handlebars(str)
  str.gsub(/{{.*?}}/m) do |match|
    code = match[2...-2]
    silent = code.end_with?("#")
    result = instance_eval(code)

    (result unless silent).to_s
  end
end

 Wow! That code is MUCH better looking! What does reek think about it?

0 total warnings

No more code smells found, well at least by the reek tool. I did make one further change. I corrected the ambiguous top comment line:

#Process a string with backslash quotes and code embedded 
#in handlebars.

It would seem that even automated code scanning tools do not check for poorly written comments.

So, I think it is pretty clear that the new code is much better than the original. I can tell you that it also runs faster and uses less memory. Can we draw a conclusion from all of this? How about:

When things smell bad, put away the scented air spray and just clean up the mess!

Best regards;

Peter Camilleri (aka Squidly Jones)

Notes:

  1. In the quest to write better code, I am inspired by Sandi Metz. An awesome video by her on the matter of smelly code is Get a Whiff of This by Sandi Metz.
  2. Some code was slightly reformatted to fit into the blog post.

 

 

 

 

The best news I’ve heard in a LONG time!

Hello All!

One of the most pernicious and damaging mistakes, injurious to world progress, has to have been the decision to allow the patenting of software. These patents have served to obstruct progress, destroy innovation, and fatten the wallets of lawyers and greedy patent trolls.

Now, to some, software patents are an awesome idea. If your goal is to provide more work for legal departments at large companies or you need to attack a competitor with the gaul of being better than you, then software patents are for you! They’re also great at destroying small, upstart companies with all of their “new ideas” stuff. Heck, even if the patent does NOT apply, prolonged legal proceedings can serve just as well.

Seriously, software patents are of dubious virtue at the best of times. Now finally, after many years of suffering under the lawyer’s savage yoke, there may be some hope:

Here’s Why Software Patents Are in Peril After the Intellectual Ventures Ruling

This a great article with a video for the tl;dr; crowd. Still, I’d like to focus on a little excerpt from the article that is, I think, the heart of the matter:

Pointing out that intellectual property monopolies can limit free speech, Mayer notes that copyright law has built-in First Amendment protections such as “fair use” and that patent law must include similar safeguards. He suggests that the safeguard comes in the form of a part of the Patent Act, known as “Section 101,” which says some things—including abstract ideas—simply can’t be patented in the first place.

I really hope that this is the beginning of the end for software patents! I really do! I look forward to the day when they and the profits of the trolls are all dead and buried. Just don’t expect me to bring flowers to the grave site.

Best regards;

Peter Camilleri (aka Squidly Jones)