Tag: Linux

It’s 2012, and the Embeddable Linux Kernel Subset (ELKS) is back!

A long time ago, some Linux kernel developers forked the Linux kernel (in truth, it was more “rewrite” than “fork”) into a version that would run on 8086/8088-class processors, like in the original IBM PC and PC/XT. That kernel was coined the Embeddable Linux Kernel Subset, or ELKS for short. The project has seen very rocky times in the 2000s as developers gradually moved away from the project, interest slowly faded, and perhaps the near-death blow for ELKS was the tragic loss of Riley H. Williams, whose contributions of time and skill kept ELKS going for years until his death. (Rest in peace, Riley; we miss you.) After Riley died, no one who was involved had the time or energy to steer the ELKS project anywhere. I released a broken tarball of all of the source code contained in the ELKS CVS database on SourceForge, dubbed it release version elks-0.1.3, and hoped that someone else with more time and knowledge than myself would be inspired and take the initiative to get the ball rolling again.

Other than an occasional blip on the ELKS mailing list (linux-8086 at vger.kernel.org), nothing really happened. I tried to spark some conversation about where ELKS should go, but almost nothing came of that effort either. ELKS was doomed to obscurity, and even I mostly forgot about it as I was distracted by other things in my life. And then, something changed!

Juan Perez-Sanchez posted a patch to linux-8086 to fix compilation errors on 2012-01-10. I starred it but didn’t really think much of it, because I was very busy that day, and I forgot about it. Juan, though, wasn’t kidding around. Over the remainder of January of 2012, Juan posted four more patches to the mailing list to fix a variety of small issues. Hans posted that “it is good to know that ELKS is not completely abandoned,” and after seeing that message in my inbox on top of all these patches, I couldn’t keep myself away any longer.

I transitioned the code on SourceForge from CVS to Git on Feb 5, and the next day I imported the CVS commit history into Git as well. I’ve decided to fully take the reins, suck it up, and become the leader of ELKS, particularly since I am the only administrator left on SourceForge who is active. So far, there has not been a single day since 2012-02-05 that the list hasn’t had a post. I’ve begun digging up some old hardware to test on, and I can safely say that ELKS isn’t dead.

ELKS is back. Anyone who wants to help should subscribe to the linux-8086 mailing list at vger.kernel.org and contribute!

Tritech Service System 2.1 progress

So far, only a few bugs remain in TSS 2.1, which is currently at  version 2.1-alpha5.  There are some problems currently being worked out with KMS support, which is the biggest issue so far.  The entire “init” system has been rewritten to replace traditional “init” with the runit-init tools provided by BusyBox.  The move to a partially modular kernel has been done, and we’re testing that out on machines to make sure it behaves as expected.  (Modular support is necessary for drivers that must load firmware, like most wireless network adapters and some of the KMS video drivers in the kernel).

We haven’t set up the “beacon server” features yet, mainly because some security issues need to be addressed.  Persistent home support has been thrown out for now, since we will be replacing it with something more robust in the future.  We’re not too far from the 2.1 release.  It’s going to be pretty sweet!  Stay tuned!

Update (2011-04-08): Up to -alpha7b2, trying to help the Linux kernel developers with some serious framebuffer issues.  It seems that Intel i915 framebuffers (inteldrmfb) like to cause a completely black screen at boot time on plenty of computers (and that’s even using today’s most recent git pull of the kernel code).  Radeon and nVidia once in a while has the same issue.  In the meantime I’ve built a second kernel that has all the graphics stuff stripped out so that we can use console-only mode.

Sort compressed tar archives to make them smaller… 20% smaller!

How would you like for your file archives to be 20% smaller, with just the tools every Linux distribution already provides and a little ingenuity?  Read on and see how I did it!

There was a folder “NESRen” that I wanted to pack up for archival, and I knew it contained many files that share the same data, minus a few changes here and there.  When packing them up and compressing them into a tarball archive, I knew that I will achieve better compression if these largely similar files are put side-by-side in the archive so that the repeated blocks of data “compress themselves away” and take up almost no space.  Unfortunately, the GNU “tar” command in nearly every Linux distribution packs up files and folders in the order that the underlying filesystem chooses, which is almost always unordered and not optimal for compression.

How do we make tar put files in an order which will compress better?  The answer is to use the tar -T option, which lets you feed tar a list of files to pack up.  The list is processed line-by-line, and each file is packed up in the order provided.  You can, for example, create a list of files with the “find” command, then hand-edit that list to be optimal, and pass the list to tar (you must use the –no-recursion option when creating the archive from this list since the “find” makes a recursive list already):

find folder_of_files/ > list.txt
vi list.txt
tar -c --no-recursion -T list.txt | xz > archive.tar.xz

In my case, however, the folder structure and naming conventions allowed for creative use of the “sort” command to arrange the files. Since there is one folder “NESRen” followed by a series of categorizations, followed by the file names themselves (i.e. “NESRen/World/Pinball (JU).nes”) I can do something like this to make all files with the same name sort beside each other, regardless of the name of the category directory (as “sort” with no options would do):

find NESRen | sort -t / --key=3 | \
  tar -cv -T - --no-recursion | xz -e > NESRen.tar.xz

The “-t /” tells sort to use a slash as a field delimiter, and –key=3 tells it to sort by the third field (NESRen is field 1, the folder is 2, the file is 3).  What kind of difference did that make for the size of my .tar.xz archive?  Take a look (-nosort archive created with “tar -c NESRen | xz -e > NESren-nosort.tar.xz”):

Size of each file in bytes:

212958664 NESRen-nosort.tar.xz
170021312 NESRen.tar.xz

Size of the original folder and each file in megabytes:

705M    NESRen
204M    NESRen-nosort.tar.xz
163M    NESRen.tar.xz

By sorting the files, I saw a 20.1% drop in archive size using the exact same compression method, with a total compression ratio of 23.1% versus the unsorted 28.9%.  That’s a huge difference!  If this were 70.5GB instead of 705MB and the data exhibited identical performance, the final archive would be 4.1GB smaller–nearly the entire capacity of a single-layer DVD-R in space savings, just by sorting the file names before compression.

Applying a similar sort-then-compress process to the packing of the “ext” version of the Tritech Service System, a 700KB reduction in the total size of the archive containing “ext” was seen.  Of course, this doesn’t help as much because the archive itself was already 32.7MB in size (700KB is only a 2.1% reduction) but it still means shorter load and boot times due to less overall data to handle.

Next time you’re packing a lot of stuff up, see if you can use these tricks to improve your compression ratio.

Work toward Tritech Service System 2.1

I’ve officially started the ground work on version 2.1 of the Tritech Service System.  Major changes and bug fixes that are already in the works:

  • Changing startup scripts and Busybox “init” to Busybox “runit” style: The biggest benefit of this change is that system services will be able to start in parallel, which will lower startup time drastically.
  • Partially modular kernel: Some drivers in the kernel work better as modules, and some don’t work at all unless they’re modules because of missing firmware files (specifically many wireless cards). While most of the kernel will remain monolithic, select drivers will be modularized to increase usability of the system.
  • Wireless support that actually functions: Previous versions didn’t support wireless adapters in any way that could be considered usable for most configurations. TSS 2.1 is going to include working wireless support in the kernel and supporting software.  Actual “easy” wireless configuration tools are not planned until a 3.0 release.
  • Basic support for packages: In our private TSS 2.0.7, we’ve included early support for loading packages from local media and our internal network server. What we use currently is not good enough for public release, but it does lay the foundation for that type of support.  TSS 2.1 will have a working implementation of boot-time package loading support, which allows extending the system without completely rebuilding initrd packages.
  • Native KMS driver/X.org support: Fixes the long-standing issues with most nVidia and some Intel graphics controllers failing to work with TSS out-of-the-box. X.org auto-configuration is also being implemented.  At least one test system with two video cards automatically set itself up with dual displays after adding Intel KMS and Nouveau KMS to the kernel!
  • udev automatic mount point handling: No more /mnt folder with tons of mount points for nonexistent devices! Mount points will be created more like mainstream Linux distributions, with volume label mount points in /media and device name mount points  in /mnt, all automatically handled by udev.
  • Software updates across the board: New versions of important tools such as the NTFS-3G driver from Tuxera, the latest Xine media player, and more are included with Tritech Service System 2.1. Additionally, old unnecessary packages and general junk files have been removed.
  • Improved boot times: Parallel service startup, a smaller initrd file to boot, and use of faster compression technologies all contribute to a much quicker startup time than previous versions.

Visit http://c02ware.com/tss.php periodically so you don’t miss the release!

Tritech Service System Community Release Edition 1.3 is OUT!

Once upon a time, I mentioned my custom Linux distribution that I built almost entirely from scratch for use at Tritech Computer Solutions.  I still remember a time when what we’ve come to just call the “TSS” was spouting “Version 0.1 ALPHA” and was horribly rough around the edges.  Since then, the changes to the whole mess have been absolutely amazing, and yet despite running bleeding-edge versions of practically everything, the ISO for the TSS has remained well under 50MB.  I’d tell the Damn Small Linux to eat their hearts out if it wasn’t for the fact that their distro has more programs and has a vastly different set of goals: where DSL carries the challenge to “fit as much as possible into a 50MB business card CD” and uses particularly spartan and/or aging applications to pull it off (and wow, they really have done an amazing job meeting that goal!) the Tritech Service System was built for a very different and conflicting reason: we needed a Linux distro that ran bleeding-edge stuff (especially the latest Linux kernel), fit in a tiny space, didn’t depend on boot media being present, and most importantly, we needed to run software that most live distros don’t tend to come with…and because of shortcomings in existing systems, I certainly didn’t want to remaster or add to them as the solution.

You can find out all about the distro and the details on the Tritech Service System distribution page, but some unanswered questions remain, such as “what prompted you to release your super-special super-secret Linux secret weapon to the world en masse?  What about your competitors?  Aren’t you afraid that they’ll take your hard work and use it to put you out of business?”

I’d like to tackle the competitor question first.  As I’ve stated before, Tritech Computer Solutions doesn’t have any competitors that are capable enough to be considered competitors in the first place.  Those that might be tend to be very Windows-oriented, with limited Linux skills (or none at all; look at Geek Squad’s oft-pirated MRI CD, which is an ugly Windows PE abomination that takes forever to start and makes me wretch at the mere sight of a screenshot…)  Even if a skilled competitor came along that had some Linux background, they’d still have to be willing to invest the time and energy into figuring out how to USE the TSS like we do.  The Tritech Service System replaces numerous software products that other shops have to purchase, such as Symantec Ghost Solution Suite or Acronis Backup & Recovery 10 for system imaging, or Passware Kit Windows Key for resetting lost Windows account passwords.  It even makes it possible to do things that you can’t buy software (that I am aware of) to perform:  replacing a corrupt Windows XP registry hive with a copy from a System Restore point (without using the hackish Recovery Console method on Microsoft’s Knowledge Base), checking key system file hashes against a known-good hash list to find infected or damaged system files, priming a SYSTEM registry hive with the required disk controller driver service and critical device database entry to enable booting from that controller (i.e. switching a controller to RAID mode which sometimes requires a different driver you can’t forcibly install), and much more.  The problem is that, much like a welder, you have to know how to use the tool to accomplish the goal.  Experience is why my “competition” can’t use the Tritech Service System to beat me at my own game: they don’t know how and honestly, to get where I am now requires more work than any typical computer hobbyist would ever want to deal with.  I wonder how I got this far without giving up, because it’s HARD to keep your drive when things get exceedingly frustrating.  For someone who fixes computers “good enough” to get by comfortably, the need to learn how to fully exploit such an esoteric tool doesn’t exist.  They’d prefer to be out boating.

Sometimes I’d prefer to be out boating.  Or fishing.  Or anything else.  If you think computers make YOU stress, be a computer technician and you’ll never complain about being a normal user again.

Now that the unpleasant self-promotional filth and obligatory dihydrogen monoxide humor is out of the way, let’s talk about why I’m releasing the Tritech Service System to the public, and what the difference is between what we use in the shop and what I released, dubbed the “Community Release Edition.”  The reason is simple: I worked very hard and ended up making an extremely useful tool that filled a void in the live Linux distro world, and I wanted to contribute to the Open Source community for making it possible in the first place.  The whole idea behind the Free Software Movement is that we help each other out and contribute our innovations to the rest of the community.  If I’ve made something awesome from things other people shared with me, why not share it with them in return?

Another side reason is that I’d like to be recognized for my work, and I’d like to show my fellow man that I have something positive to contribute to society.  I can’t fix a lot of the problems in the world, but if my Linux distro makes just one person’s life easier and they thank me for it, that’s all I want.  If it helps a hundred people, that’s awesome too.  If I open up donations to continue working on it and can spend a chunk of time each day devoted exclusively to improving the system because of that, I’d be absolutely thrilled and more than happy to do it.  Software work is my true hidden passion, and if I can use that passion to help others, I absolutely will.

That’s why you can download the Tritech Service System Community Release Edition.

As for the rest of the questions…the difference between TSS and TSS CRE is the exclusion of “internal use only” custom software and scripts and custom graphics that we have full rights to redistribute; also, why is the first release numbered “1.3 CRE” anyway?  Internally, we started TSS with 0.1 alpha, which progressed to 0.1, 0.1B, 0.2, 0.2A, 2B, 0.2C, 0.2D, 0.2E, then when the GUI and extended system was added we had 1.0, 1.1, and our latest internal release version so far is 1.2.  The next minor version in the sequence is 1.3, and because some minor updates exist between TSS 1.2 and the TSS CRE, I figured it would make sense to just go up a number.  Internally, there is no TSS 1.3 at all.  I literally had to rewrite 40% of the build system and shuffle tons of files around to make the build system much more customizable and robust.  TSS CRE is what motivated me to add automatic inclusion of some customizations and create a “cleanup” script.  In fact, most of the 1.2 -> 1.3 differences lie in the build system being totally different, not so much in the software that makes up the final product.  I plan to further enhance the “gentss” script to include customizations in a more flexible manner in the future.

I’m already knee-deep in the 1.4 development process too.  Tritech Service System Community Release Edition 1.4 will feature the latest versions of many libraries and packages, better boot scripts, and possibly a few pieces of software that aren’t on 1.3.  Stay tuned and see what develops.

Sylvania G’s VIA C7-M versus Windows XP

I changed my Sylvania G (original, non-Meso) netbook to Windows XP/Linux dual-boot to test some software I’m working on, and discovered that while Windows XP certainly does boot and run in general on the G, some kind of system timer or timing loop is severely out of whack! I wanted to use my little G as a portable gaming machine from the Windows XP install, and to my horror, ZSNES couldn’t decide what speed it wanted to run! Now, I’ve never had a single issue with ZSNES on any computer I’ve ever tried it on, even preferring the Windows port of it over the Linux native one, and not once has a problem existed with ZSNES that I couldn’t find an easy fix for, until now. I’ve been researching the matter and gathering evidence, and I may have a potential answer to the problem.