Hidden Files

The news is that C64 OS version 1.06 has just been released, March 22, 2024. I'm happy to get that out the door, as it frees up my mind to start working on important new cornerstone technologies coming to C64 OS. I'll start dropping hints about that in the coming weeks.

Version 1.06 is a more modest release than 1.05 or 1.04. But I think that's okay. v1.06 includes one new Application, three new Utilities and new features and improvements to several existing Apps and Utilities, and even some new low-level features in the KERNAL and libraries.

This latest release makes use of a combination of all of the above to provide a handy new feature for users and a potentially powerful and useful feature for developers, when put to creative uses at a low-level. Discussions of just this nature have already been spurred on in the developer forums on the C64 OS Discord server.

That feature is: Hidden Files. (But you knew that already from the title of this post.)

Let me give a quick walk through of how hidden files work in C64 OS, some of the pit falls, some of the advantages, etc.

If you haven't yet, go grab the free v1.06 update here.

What does the storage device do?

On the Commodore 64 and other 8-bit Commodore machines the storage devices themselves take on the burden of file management. Let's review what various pieces are handled by the storage device itself. We'll use the 1541 disk drive as the canonical example.

It has a 6502 CPU, a 16KB ROM chip and rather modest 2KB of RAM. It also has two MOS 6522 VIA chips. The Versatile Interface Adapter (VIA) is essentially a CIA (Complex Interface Adapter) like the ones found in the C64, but with some features left out that a disk drive doesn't need.

It's what exactly the 1541's CPU and VIAs do that is interesting. They implement almost everything right down to the nuts and bolts.

The 6502+VIAs together, in the 1541, control the position of the read head and when to spin the disk. (Although not how quickly to spin it, that part is fixed at 300 RPM.) They perform the task of reading one bit at a time off the surface of the disk as it spins past the read head. The bits read are in a special 5:4 ratio GCR encoded format. To read a block of data, the 6502+VIA reads one sector of 320 bytes of data (2560 bits) into a RAM buffer. The 6502, running code stored in ROM, then decodes the 320 bytes of GCR data into a block of 256 bytes of usable data. In other words, the 6502 and at least one of the VIAs, running the the code in ROM, serve as the primary drive controller that reads and writes bits to and from the physical media. That's very low-level.

It doesn't stop there though. The ROM chip also contains code that gives the 1541 an understanding of the file system. In other words, when you load a directory, the computer sends only a single very simple command to the drive. The 1541's software in ROM knows where the first directory block is located and moves the read head to the right track (track 18.) It knows what is the first sector to read, and it proceeds with the low-level operation of reading in that sector and decoding it into a block of data.

Additionally, the ability for the computer to make requests to the drive at all is because the 1541's 6502 plus its second VIA, again running software stored in ROM, knows how to communicate using the IEC serial bus protocol. The 1541's ROM software implements the concept of 16 channels (0 to 15) which work in tandem with the IEC bus protocol. It is the 1541's ROM software that makes it know that channel 0 is for loading binary data (or the directory) and that channel 1 is for saving binary data, and that channel 15 is for communicating commands and status data.

When you request to open a file, it's the drive and its ROM software that know how to find the directory blocks, know how to move the read head, know how to read in the GCR data one bit at a time, know how to decode the GCR data, know how to interpret the directory data and search for a matching filename, and when found, know how how to open the file by locating its first data block. The drive knows how the first data block is linked to the second data block. The drive buffers one block of data at a time in its RAM, and knows how to spool the data out to the computer one byte at a time, one bit at a time, over the IEC bus implementing the bus protocol with 6502 instructions controlling a VIA chip.

Hot damn! I hope you're starting to get the picture here. The software in ROM is executed by the 6502 to command the VIAs to do literally EVERYTHING. It's incredible! What's even more wild is that the drive encodes a directory listing in the format of a BASIC program so that it is loaded into the C64's memory the same way as any other program, and listed just like any other BASIC program has a listable set of lines.

Potential downsides to the Commodore 8-bit way

What this means is that the Commodore 64 itself basically has no idea what it's doing.

When you open a channel to a device on the IEC bus, it's generally a storage device, but it doesn't really have to be. We could be living in a world in which a whole series of desktop peripherals communicated over IEC, like flatbed scanners, cameras, or sensor equipment. We already have printers and plotters but basically any type of device that takes or provides data could be run on the IEC bus. As long as it makes sense for the computer to be the bus master, the one who calls the shots about who should be using the bus and when.

It is this agnostic approach that allows cool projects like Meatloaf to be possible. The computer thinks it's opening a channel to a device on the IEC bus (which it is) and meanwhile the device is fetching data from the internet over a WiFi connection. The C64 has no idea. That's brilliant.

The downside is, the drive is in 100% control of how it all works. Why is this a downside? Because it puts the onus on each drive manufacturer to implement their drive's behavior in a way that is consistent and compatible with other drives. Commodore was very careful to maintain a strong family resemblence between the devices so that the behavior of a CBM 2040 and a CBM 4040 was carried forward faithfully to the 1541 and the 1541-II and the 1571 and the 1571-D and the 1581, and all the minor revisions in between.

What would happen if some drive manufacturer said, "You know what? When we return a directory we're going to structure it in some totally different way." There is nothing preventing the drive from doing that, and nothing the C64 itself can do or say about that. The only thing discouraging drive manufacturers from doing that is the abstract social pressure of the marketplace. For example, if typing load"$",8 resulted in something weird and unexpected, that would annoy people, so they might not buy your drive, even if you had good reasons for the change. Only this basic market pressure compels manufacturers to conform to the de facto standards.

Normally changes would not be quite that dramatic, but they could take the form of failing to implement certain commands, or implementing commands that work in an unexpected way or take arguments in a different format. Here's a concrete example; IDE64 has no copy command! What?! How can that be? But it's true. It has no copy command, and the User's Guide just silently doesn't talk about it. The standard copy command uses the following format, sent to the command channel (Channel 15):

c:newfile=oldfile

But if you issue this command on an IDE64, it just says it's a syntax error. I guess they assumed that if you want to copy files you would use the built-in file manager software. But this means you cannot programmatically (i.e., from your own software) command the IDE64 to make a copy of a file. Its DOS just does not know that command. And there is nothing you can do about that.¹ That is most certainly a downside.²

How is it different on other computers?

This division of labor might have been a specialty of Commodore in their 8-bit days. The Amiga does not work like this, nor do any modern computers. It isn't just about its age though; I believe the Apple II behaved more like a modern computer than like the Commodore 8-bits.

In a modern computer some things are the same. The storage device knows how to speak its communications protocol. That might be USB or Firewire, or SCSI, or IDE/ATAPI, or eSATA, etc. Additionally, the drive has the ability to read and write the data to and from the physical media, with whatever sort of encoding system is necessary for error correction (that's basically what GCR is about.) However, they typically abstract the media into a large number of sequential blocks. And then the communications protocol offers a generic set of commands to read or write blocks by number.

The legacy Commodore disk drives, plus the CMD drives, and even IDE64, also provide block level commands. The Commodore disk drives don't, however, abstract the blocks into a sequence. They require you to specify tracks and sectors, the available number of which of course varies from drive to drive. We can think of this as an incomplete abstraction. The CMD devices and the IDE64 took the next step of abstracting the sectors within a partition into a single series of sequentially numbered blocks. On the other hand, SD2IEC lost this ability altogether, outside of mounted disk images.

On other types of computers, then, we have the following situation: The drive handles reading and writing blocks but the file system that makes use of those blocks is implemented in the computer's memory, managed by the computer's CPU, under the direction of software drivers. The software drivers understand how the directory is structured, how it makes use of blocks, how it connects directory entries to files, how blocks are allocated for files, and so on.

The software asks nothing more from the drive than to read and write numbered blocks of data. The drive is the dumb party in this setup. The drive has no idea what the data in those blocks means. Usually there is some sort of basic file system support in firmware to boot strap the machine (BIOS, EFI, etc.) Once it starts booting up though, the software drivers take over. This requires more from the computer. More work has to be done by the computer's CPU. More memory has to be occupied. And both memory and CPU time are in very short supply on an 8-bit Commodore.

What is so powerful about the modern way, though, is that a new file system can easily be applied to older physical devices. Devices that were invented earlier than the existence of newer file system innovations can usually be formatted down the road to use the newer file system. And that means older devices can get the same set of features as newer devices, as long as they get formatted with the newer file system. In other words, you can take a crappy old USB stick that came with a FAT16 file system, and you can format it with a journaled and encrypted APFS with multiple virtual data pools, simply by plugging it into a modern Mac and formatting it with the most recent version of Disk Utility.

This is not the case on the Commodore 64. So, where does that leave us?

Inconsistency between drives sucks.

Different file systems provide different attributes that can be applied to files in a directory. For example, on the 1541, and similar Commodore disk drives, a file can have a CBM file type (PRG, SEQ, USR, REL, etc.) A file can also be locked, preventing it from being scratched. A file can be marked open, preventing it from being opened more than once, and also indicating if a file was improperly closed (the famous splat files requiring disk validation.) Files can have a 16-character filename with some restrictions, but not many, on which characters can be used. And on modern devices a date/time stamp when the file was last written to can be saved along with it.

Other file systems provide options for other attributes. The file system on the IDE64 supports a hidden attribute. SD2IEC, which is backended by a FAT16 or FAT32 file system, also supports a hidden attribute. FAT32 supports a readonly attribute too. That's all pretty cool. You can hide files on an IDE64 and on an SD2IEC.

The problem is... hiding files doesn't work on a CMD HD, nor a RAMLink, nor an FD2000 or FD4000. And it certainly doesn't work on a 1541, 1571 or 1581. What about on a Meatloaf? Who knows. It would depend on whether they've implemented that feature, and the commands to toggle the hidden status of a file. What about on an SD2IEC when you mount a .D64 disk image? How can it represent a hidden file when that legacy file system is back in play?

And this generally just sucks. Inconsistency sucks.

Why does inconsistency suck? Let's face it, this stuff only starts to matter when you start getting serious about using mass-storage devices. But we're talking about C64 OS here, and it is, if nothing else, serious about using mass-storage devices. If you install the C64 OS system directory on an SD2IEC, and SD2IEC supports a hidden attribute, then you can start to use it and you can hide system directories, and you can start to dream up cool new things you can do, like hidden side-files that store metadata for data files, and so on.

But then someone installs C64 OS on their RAMLink, and its super fast and a very cool device in every respect, but it doesn't support hidden files. And suddenly whatever cool stuff you were dreaming up that would work on SD2IEC or IDE64 doesn't work on a CMD HD or RAMLink or perhaps some other mass storage device that will pop up. And that... just... sucks.

Hidden files, a feature of the file system?

The way Unix hides files is by putting a dot (.) as the first character of the filename. This might be an urban legend, but apparently the feature began as a bug in LS (the directory listing command.) Because the directories have "." and ".." entries which are used to refer to the current directory and the immediate parent directory, respectively, the LS command wanted to omit these from the standard listing. It did so by skipping over a file if it started with a dot. This had the side effect of skipping over all filenames if they started with a dot. As it turns out, people liked this and considered it a feature. So LS only includes files that start with a dot if an argument (-a) is used to explicitly include them. And hidden files were born.

There are upsides and downsides to using a leading dot to indicate a hidden item (i.e., file or subdirectory.) One upside is that every file system (on the Commodore, at least) lets files have a name that starts with a dot. So the "hidden" attribute doesn't have to be a special extra attribute supported by the file system, and it doesn't require the drive to have special commands to toggle that attribute. It's just part of the name.

One downside is that, again at least on Commodore File Systems, the name can only be 16 characters long. If you throw in a dot, now a filename can only be 15 characters long. You certainly wouldn't want to start encoding a whole series of attribtes in the name. Another more serious downside is that a file's name is what is used to reference it in the file system. If code is written to open and read in a file, let's say a file from the settings directory, it might have a hardcoded (relative to the system directory) path of "/settings/:somefile.t" What happens now if we say, "You know what would be cool? It would be cool to hide the settings directory, and just clean up the main system directory for the normies." But if doing that requires renaming the settings directory to ".settings" then the above code breaks. Similarly if you just, for some reason, want to hide that one file, if you have to change its name to ".somefile.t" suddenly the reference would have to be "/settings/:.somefile.t" and that original code still ends up broken.

If we accept these downsides, then at least future files and subdirectories could be created as hidden right from the start. Another upside is that the dot is the first character in the filename, and it's only 1-byte long, making the detection of a hidden item very easy to compute, as we will see.

Directory library and filtering

C64 OS v1.06 introduces hidden files, which is honored by File Manager and by the Open and Save Utilities. It does it by means of a more general feature that was added to the directory library; directory filtering.

Since the pre-release beta days of C64 OS, directories from all devices are loaded and parsed by the directory library (//os/library/:dir.lib) into linked-page standard directory entry structures. The first directory entry that needs to be loaded allocates one full page of memory from the paged-memory allocator. One directory entry uses a 32-byte (aligned) chunk of that page. 256 bytes in a page, divided by 32 bytes per directory entry, gives room for 8 directory entries per page. The first byte of which is not used.

The directory library continues to load in entries, filling in 32-byte chunks in the current page, until the page is full. As soon as it needs room for a 9th entry (or a 17th, and so on,) it allocates one new page, and the new page byte is filled into byte zero of the preceding directory page. This is cool. Directories become a chain of linked pages, and the full directory doesn't have to be stored in one giant sequential block of memory.

I've been thinking a bit about how high-level APIs on modern computers are able to open and read through a directory so easily. And I've been pondering, how do they do this, and is the way that C64 OS does it just overwrought? For example, here's how you open and scan through a directory in PHP:

$dir = opendir("."); //Current directory

while($entry = readdir($dir)) {
	//$entry is a just a filename.
}

closedir($dir);

It's pretty damn simple, right? Actually, it's just the nature of a high-level language, wrapping and hiding the underlying complexity. Could I write a library that implements opendir, readir and closedir that works basically the same way? Absolutely. It would just hide everything about the way it currently works. The way dir.lib works (which is a healthy balance between low-level and high-level, in my opinion,) you have to provide it with a fairly large and complicated, 62-byte directory metadata structure. Like this:

Certain properties can be set in that structure to inform dir.lib how to make the request, and certain properties are used as a place to store data and properties about the directory itself, which is not just found on any particular directory entry.

If I were to implement opendir/readdir/closedir, I would have opendir allocate the directory metadata structure and pre-populate it with standard values. Then readdir would move an index through the directory entry structures and return a pointer just to the filename component of each entry. And lastly, the point of closedir would be to give it an opportunity to free the memory: free the linked directory page structure and free the directory metadata structure. Some other high-level routine, like dirinfo, could be used to abstractly probe the metadata for information about the directory, such as how many files, or the sum of their sizes, etc. Creating that higher level directory API is an exercise for some future date.

The directory metadata structure has a field for a file matching pattern. By default it is configured as just "*". The DOS on every device recognizes "*" as a pattern that matches all entries. You can change this pattern, either programmatically or let the user do it by providing an interface to let them type in a custom pattern. dir.lib knows nothing about how the pattern works or what it means. The pattern is sent to the device, and the device interprets it and limits the directory entries it returns.

There is a problem though. As discussed at length earlier, what patterns a device supports is up to the device, and it's inconsistent; they're not all the same!

You may have noticed that some Commodore 64 software prefers to use prefixed filename extensions. (C64 OS and its libraries support both post- and prefixed extensions.) GoDot, for example, has a series of files like ptr.* for pointer files, svr.* for savers, ldr.* for loaders, etc. The reason is because on a 1541 all characters in a pattern following a * are ignored. If you load a directory like this:

load"$:*.koa

It doesn't just find files that end with .koa, it matches everything. If you load a directory like this:

load"$:ldr.*la

It doesn't just load the files that start with "ldr." AND end with "la", it loads all the files that start with "ldr." and end in any way. The "la" is ignored. Of course later devices don't do this. CMD HD and RAMLink and SD2IEC, etc. do what you would think they should do in the above examples.

There are things the patterns can't do though. For example, the following:

load"$:a*b*.sid

I'm not entirely sure what's happening under the hood here, but neither CMD HD nor SD2IEC is able to match a file called "appleblossom.sid" using the above pattern. My guess is that it doesn't expand the second *, and instead tries to match it as a literal. But since * is not legal in a filename you're never going to have a file with a literal * in the name, so it just doesn't match anything.

Those shortcomings aside, there are other things that the pattern cannot even attempt to do. For example, there is no way even to express the idea that it should match all the files whose first character is NOT something or other. This is stuff that the Amiga file patterns started being able to express, and it definitely makes the Amiga's command line much more capable than what you can do from the C64's READY prompt.

Prior to C64 OS v1.06, the directory metadata structure is 62 bytes, and the last byte is for directory special flags. The dir.lib automatically identifies the special system directories: //os/applications/ and //os/utilities/ and sets a flag for those. This allows File Manager to treat bundles like single files rather than like directories, and also to show the custom icons on the files in those directories.

Another special flag indicates if this directory is capable of supporting subdirectories. dir.lib knows if the directory is a mounted 1541 disk image on an SD2IEC, and clears the flag indicating subdirectory support. This allows File Manager to grey out options for creating a new subdirectory here. Similarly, if the directory is the partition directory of a device, dir.lib flags that. File Manager uses this flag to know that it is not possible to create directories, create regular files, or copy/move files here, nor can you scratch a partition.

Prior to 1.06 there was one special flag that the program sets, TS_TIME, which tells dir.lib that it should (if possible) load in the timestamp data with the directory.

Starting from 1.06, there is a new flag: TS_FILT. If this flag is not set, everything behaves exactly as it did before. So, existing uses of dir.lib don't have to change a thing. However, now, if the TS_FILT flag is set, dir.lib expects the metadata structure to be 64 bytes instead of 62. The extra 2 bytes at the end are a pointer to a directory filter routine. Previous code that allocates memory for just a 62-byte metadata structure don't have to worry about the 2-bytes immediately following that structure. If TS_FILT isn't set, dir.lib will not touch, modify, or dereference those final 2 bytes, so they are safe to be something totally unrelated to the dir metadata structure.

The directory entries returned had to match the pattern first, so you can combine a DOS file pattern with TS_FILT. If TS_FILT is set though, after each patttern matched directory entry gets loaded in, parsed and stored in the directory entry structure, then a call to the filter routine is made. In the filter routine, frefptr, deptr and mdptr (as defined by //os/s/:dir.s) are set and can be used to analyze the entry that was just read.

frefptr points to the file reference where this directory is found. So you could use this to make global filtering choices based on the type of device or the path or partition number.

mdptr points to the directory metadata structure, which is being updated as entries are read in. So you could use this to make filtering choices based on anything in that structure.

But the main driver is deptr. deptr points to the directory entry structure that was just read in. You can use that to reference any property of the file; its CBM file type, its lock status, its size, its datetime stamp, or, of course, its filename. The world is your oyster. You can combine any criteria that you can programmatically express. Like, only PRG files that contain a letter "b" but not a letter "a"? What you need, you can do.

The filter routine returns with the carry clear to accept the entry and move on to the next. If the filter routine returns with the carry set, the directory library simply deallocates the entry just parsed and stored, then moves on to the next. The next entry read overwrites the place where the filtered entry had previously been deallocated. It's that simple.

Putting directory filters to use

There are two places that have nothing to do with hidden files, per se, where I immediately put directory filtering to use.

As a developer, my utilities directory (//os/utilities/) is full of source code files and label headers mixed in with the actual launchable Utility files. It has always bugged me that when I open the Utilities Utility it lists the source code. It's especially annoying because if you accidentally try to "open" one of the source files as a Utility, it crashes C64 OS.

Adding a filter on this, the first thing it does is omit any file that isn't PRG. That very efficiently skips header files. Next, it scans the filename for a dot ("."). The first dot it finds causes it to return with the carry set, without looking any further. This gets rid of all files with an extension of any kind. Real Utilities should have simple human-readable names, such as:

• Calculator
• Colors
• D&D Roller
• File Info
• Mouse
• Opener, etc.

Will this new filter hide a legit Utility called "Mr. Scheduler" ? Yes. But, now you know why and you should avoid using a dot in the name of your Utility.

The Opener Utility, when it shows its UI, has a list of Utilities and a list of Applications. The same filter was applied here for the Utilities. And for the Applications, everything that is not a directory is omitted, however, this doesn't require a filter. In this case you can set the file type (td_type) in the directory metadata structure to only include directories ("b" for branch. Ask CMD why it's not "d" for directory, don't ask me.) If you can use the DOS's pattern and type matching, you should use that first since it's more efficient. But when you need filtering, it's available.

Lastly, hidden files. File Manager, Open and Save Utilities, all implement them the same way. There is a single filter routine that is super duper short. It looks like this:

hidefilt ;Hide files filter
         ;deptr -> dir entry
         
         ldy #fdname
         lda (deptr),y
         cmp #"."
         beq done
         clc
done     rts

Whenever the result of a compare instruction is equal, the carry is set. If the first character of the filename is a dot, it branches to RTS and the carry is already set, and that entry is omitted. If the compare fails, the carry is in an indeterminate state, depending on whether the first character is less than or greater than the PETSCII value of a dot. So it explicitly clears the carry and falls through to RTS to accept the entry. And that's it for the filter routine.

When the user chooses to show hidden files, the TS_FILT bit is cleared in the directory metadata structure. To hide files it just flips that bit on and the filter routine starts being used. Also, because every tab already maintains its own directory metadata structure, files can be shown or hidden on a tab-by-tab basis automatically. And, because File Manager was already set up to save and restore the settings of the dir metadata structure, it automatically retains your preference for showing or hiding hidden files. The implementation was so beautifully clean!

Final Thoughts

I hope you found that interesting. Hidden files in C64 OS makes it feel more modern. I've already put them to use in the Memos Utility, new in v1.06. Typically first-party Utilities store their data somewhere in the system directory. But, rather than just adding a new "memos" directory, which over time would make the system directory gather more and more stuff, I called it ".memos". It's still there, it just doesn't clutter up the system directory.

The backup tool (//os/c64tools/:backup) also writes its last run timestamp to a file called "backup.ts.t" into the root of whatever directory you backup. I often run it on the system directory itself, but once again, that leaves an extra file just cluttering up the system directory. The tool uses this file for knowing what it needs to copy when doing an incremental backup. Starting in v1.06, it writes the file as ".backup.ts.t" and checks for that when it starts an incremental backup.

What ideas will you come up with for how to profitably use directory filtering and hidden files?

1. It's not that you can never programmatically copy a file, but you have to implement it by reading from an open file on one channel and writing to another file on a different channel. Given how IDE64 works, this probably isn't substantially slower than a built-in copy command would have been.
2. You cannot, for example, initiate a file copy from a BASIC program. And BASIC is too slow to implement the copy routine manually.

Do you like what you see?

You've just read one of my high-quality, long-form, weblog posts, for free! First, thank you for your interest, it makes producing this content feel worthwhile. I love to hear your input and feedback in the forums below. And I do my best to answer every question.

I'm creating C64 OS and documenting my progress along the way, to give something to you and contribute to the Commodore community. Please consider purchasing one of the items I am currently offering or making a small donation, to help me continue to bring you updates, in-depth technical discussions and programming reference. Your generous support is greatly appreciated.

Greg Naçu — C64OS.com

Want to support my work?

C64 OS version 1.04 is now available

The best way to help C64OS.com grow and continue is to purchase a commercial copy of C64 OS version 1.04.

Join the growing C64 OS user community. Take advantage of the powerful and useful features available now, and get access to run new and future software made for C64 OS.

Learn More Order Now

C64OS.com is hosted on a Digital Ocean droplet.

Digital Ocean is a cloud hosting platform, built for developers. They make it simple to get started. They have intuitive administrative tools and reasonable prices. And they can scale as your needs grow.

If you sign up with this referral link, you'll get a $100 credit, and after you've spent $25, I'll get $25. If you are looking for a cloud platform to host your website or web app, this is win-win.

Sign up for Digital Ocean with this refereral link.
Save money and help C64OS.com at the same time.

No, thank you.

Want to support my hard work? Here's how!