Programming Language

Binary Representation of Source Code

• Represents the source code. Is not any kind of executable code (ASM/bytecode). Not an IR.
• Take any valid text source code, turn it into the binary representation and back again and end up with the same byte for byte file.
• Not storing individual token (ie no LEFT_BRACE). But do need to keep things like whitespace and comments.
• Edit source code not text.
• But still allows for people to use standard text editors.
• Also allows for non-text sourcecode specific editors.
  • Quick and efficient editing of the binary format (ie quickgo/quickrust concept programs).
  • Graphically represent source code (not the same as a graphical programming language, ie blocky, just an eaiser way to see read code).
  • Having things like frames around things like data structures and function definitions.
  • Could have UML like representations (Not advocating for UML specifically, but it's a possibility).
  • Easy/quick navigation of source code. Things like goto definition would be much easier to represent.
• Makes tooling much easier. Can allow for libraries for manipulation of the code that tooling can use.
• Down side, any time you have an invalid syntax everything breaks. But that happens anyway with normal code...
• Could use a virtual filesystem to automatically convert stored binary to text or visa versa.
• Any text you edit could basically have any syntax you like, although obviously a standardised version would be best.
• Could allow for syntax changes.
• Could allow for special keywords for editing with a basic text editor (ie 'def myfunctionname' could be hooked to actually insert a function definition nearby on file save and the 'def' keyword removed).
• Would be easier with a well defined syntax for the source code (ie define tabs vs spaces, number of newlines between functions).
• But might be better to just store tabs/spaces and newlines in the binary format.

Schemas not 'data structures'

• struct definitions are normally mixed in with the the procedural instruction source code.
• Structures are a binding of **data types** to **variable names**.
• Separate the **representation** from the **implementation**.
  • Standard native in memory with the same performance and so on.
    • Allow for separate memory layout. Some arch (for example Cell processors require memory padding).
    • In memory ordering.
    • Endianness?.
  • Serilization.
  • Database backed.
• Older OOP languages like C++ and Java also bind **methods/member functions** to **data structures**.
• Newer languages like Rust and Go move away from OOP and use interfaces (ie traits) primarily.
• Conceptually design it as **API's** not bound functions.
  • Allow for standard native function calls, or RPC calls, etc... IPC or networked.
  • Some kind of distributed backend (Raft, Blockchain)
• Could allow security definitions in the schema. Ie, who can edit this variable. Allows you to separate the security implementation stuff from the data structure.

API Versioning

• API version as a hash of the binary representation of the API?
  • Need to deal with non-breaking things. Like changing the order of functions.
• Function definitions and the like could be tracked, and breaking changes to syntax noted automatically.
  • Allow adding fields with defaults without api change.
  • Allow optional named arguments.
• Implementation stuff is harder (ie we changed the format of the string this function returns but the signature is the same).
  • Functions that have no source code changes can be safely ignored.
  • Changing the implementation doesn't mean the result is different (ie optimisation).
  • Changing the implementation of a function could accidentally change the result (bug). Being told when that happens is handy.
  • Allow specifying functions for specific API versions so if you do change the implementation you can keep backwards compatibility.
  • How do consumers choose which version (ie specific version they used, or 'latest'?)... Compiled binaries could keep a list of the api version used.
  • Unit tests could provide a hint. (ie if this unit test changed...), but doing something like adding an extra test or changing the order doesn't mean the implementation's result is different.
    • Automatic 'quickcheck' when possible? Compiler can implement a unittest with no effort from the programmer and log results. But you won't know when it's possible (ie halting problem, use of globals/statics, side effects, etc...). Maybe just best effort (ie if it didn't finish in 1 second and/or used more than 512kb of ram, kill the test). Don't store the result of tests that returns a lot of stuff. Do store the meta information about killed tests and the number of items returned (or even better a hash of the items returned, pointers would be a pain though...).
    • 'quickbench'? To benchmark performance? Obvious problems of different hardware but could still be useful. Probably not for API versioning

Everything works as a module/library

• Main is just a function gets passed an os.args, stdin, stdout, etc... into.
• Have a 'Enviroment' struct/context that has all that in it. Could make an implicit 'global' one but allow it to be overridden in functions. (ie remap printf to fprintf). Does that effect performance, how to deal with compiled libraries? Don't want extra bytes getting passed to every function or an extra redirect on thing like print statements... Look at how printing happens in ASM... Maybe printf just remaps to fprintf(stdio...)
  • stdio
  • stderr
  • ENV variables
  • filesystem,
  • Operating system functions...
• Could sandbox things by just removing things like the os.filesystem (unless you can somehow manually do a function pointer to it).

Everything as an interface

• For example, file access.
  • No global fopen("filename");
  • Instead open(os::filesystem, "filename"). Although a os.open wrapper could be used for the lazy, maybe it should be avoided since it's use should be discouraged, especially in libraries. You don't want to use a library that forces a config file to be stored in a specific location when you want to use a database as a backend store for configuration files.
  • In many ways a file over a network connection is the same as a file on a harddisk.
    • A hdd can die, become full or be removed. A network cable can be unplugged (plus the files on the other end are going to be stored on a harddrive anyway which have the same problem).
    • Differences?
      • Metadata stuff.
        • Rename files, can't normally rename HTTP. Renaming doesn't effect the actual file, it effects the filesystem's index.
        • Linking files. Can't link a HTTP file, locally (well not without the OS doing it, or a virtual filesystem library but that's out of the scope of a programming language). Once again not about files, but about the underlying filesystem.
        • Timestamps, ditto.
        • synchronisation and atomic operations. A file stored on a disk can be fsynced so you know it's stored. An atomic operation can allow a file to be replaced in place. A database on the otherhand might be holding the file in ram.
        • Files are lockable. Prevent multiple processes trying to write to the same file at the same time.
• println("This is bad as it's pain to override and isn't testable...");
  • fprintln(stdout, "This is better but more typing");
  • Monkey patching can work but deals with globals which adds threading issues...

Stop allowing stuff in 'empty space'

• Syntactically quarantine globals/singletons/statics if they are allowed

globals {
   int hello = 0;
}

• Do need things like function calls and so on.
• Don't syntactically need to specify the module namespace if it's determined by the filename/location.