For those who don't know SkyAlt. It's the software from which you connect to remote databases(or REST APIs), create user-friendly front-ends and get insights through reports and charts.
I've worked on SkyAlt for more than a year. I released two builds, everything looked good, but 2 months ago, I run into a really big problem. Too many bugs, too many lines written in un-safe C language. I've programmed for almost 20 years, I shipped the products, but SkyAlt is the most complex project I've ever created it. I could use the common strategy to add new features and then keep testing it until it looks stable and hopes it's actually stable, but my goal is to build the best product on the market.
So the new goal was to express what I want from the computer in the most efficient way and avoid bugs which are harder to find. This happens to me from time to time, when I'm lost and I tried to rethink the whole thing from scratch and in most cases I end up with what I have. So basically, I waste my time. But sometimes big risk brings a big reward.
The solution was obvious, a new language! Less accidental complexity. C is not great for large codebases, but it's a good place for creating fast interpreters. Have you heard about DSL(Domain-specific language)? I did, many times, and I ignore it every single time. But not this time.
So I paused adding new features to C code and started replacing a big part of the original code with less code in the language which I'm going to invent.
The First 3 days I had these crazy ideas about new programming languages, something with a declarative paradigm that would require some constraint solver to execute only what It needs. I realize that it would take months/years to build something like that and I wanna keep my "intermission" short as possible, since I'm running a startup.
It took one A4 paper and 1 m^2 of the white board, the rest was my imagination to came up with the language to significantly simplify my previous work. The syntax is very expressive for my use-cases(organizing and analyzing data - relational databases), with minimum accidental complexity.
I was highly influenced by Bret Victor's Drawing Dynamic Visualizations.
The code is procedural "tree" with permanent storage. So SkyAlt executes the whole tree and then it deletes all temporary data(variables in functions/blocks) and starts executing the whole tree from scratch. If you want to keep some data to the next "tick" you save it to permanent storage. This approach is really great for debugging(catching bugs quickly) since most of variables are reset after every "tick".
Try to imagine something like Immediate GUI, but I don't use it only for front-end(drawing Buttons, Combo-boxes, etc.), but also for the back-end(db connectors, filters, etc.). Otherwise it's dynamic typed, safer(no out-of-memory, malloc, free, etc.), easy to work with strings, ready for translations to foreign languages and much more. On the C side, it's just parser which creates the AST graph and then the interpreter goes through the graph and executes nodes.
In the beginning, porting SkyAlt was really slow, because often I had to go to C code and repair bugs in parser or interpreter, but later I spend most of my time writing code in my new language and I was blown away how fast I can actually add a feature. Fun.
There were days when I fixed bugs.
There were days when I changed the syntax.
There were days when I added new features.
There were days when I rewrote some parts from scratch.
In the end, I slept 1-2hours/night less, drink twice the amount of coffee and successfully ported 80-90% of original SkyAlt C code. It runs a little bit slower, but it's very stable!
Comparision of SkyAlt build 2.5(all C) and SkyAlt build 3.0(C + new lang)
I thought about rewriting SkyAlt to Rust or Swift. But I really doubt that I would get 10x less code. C/C++, Java, Python, Rust, Swift are general languages that cover a very high range of use-cases and that's also their weakness. Sometimes things can become better with an army of macros(or templates), but I've never liked doing that to C.
When you work with general programming languages, you probably write open() and save(). In SkyAlt, there is no need to write code for reading/writing projects to file. There is de/serialization build in.
Many programmers have struggled with the progress bar. It's tricky, there are so many places in programs that may take longer to execute and they should tell users how long it will take. In SkyAlt you don't create progress bar and you don't update how much computing has been done. The platform will figure it out. It's not 100% accurate, but it's better than code, where programmers ignore it(frozen) or they messed it up. And of course, if you press interupt then interpreter stop.
There is also one small thing, which saves me a lot of time. Many REST APIs return JSON, so I made a function that converts JSON to object structure. In SkyAlt, every object has value and sub-objects(which have values and sub-objects) and JSON nicely translates into it. For me it's simpler to work with objects than JSON. This is only possible because SkyAlt has dynamic typing.
But the hardest part was to avoid to make the language more complicated because it will run faster, have lower memory footprint or the C code parser can be shorter. No exception, the DSL first, parser and interpreter second. I'll have years to figure out how to make it super fast.
A new language or platform is not just features, syntax or parser quality(warnings, etc.). When people talk about Doherty threshold(whatever actual number is), they mostly see it from the end-users perspective. But what about developers? Most of them go through this insane loop, over and over again:
1) write code(maybe hit ctrl+s)
2) compile + run or refresh page
3) put program to state, which you wanna test(open project, dialog, etc.)
4) search for a bug
5) close the program
and goto 1)
Crazy, right? What every programmer should do is only write code and test. And with the current pipeline I do only steps 1 and step 4, nothing else! The saved time is tremendous.
My solution works for front-end and back-end, so It's not just drag & drop GUI, but the whole program and It works for complex programs as well. There are many REPL and live programming solutions, but from what I saw, most of the developers don't use them in real productions.
For example, I open a dialog, then a context menu and then I update code for the context menu in ordinary text editor. When I save the file, the platform recognizes that source code has been changed and renders the new context menu.
Yes, It's interpreted(not compiled), so it's slower, but there are two really cool advantages.
First, users don't have to stop Skyalt process(project) to auto-update SkyAlt. Of course, this is only true if the binary file(C code) is intact.
Second, you can do a lot of things with computers, like send a message or file, but can you send a running program to another computer? Right now, You can't do that with SkyAlt, but it should be simple to implement it.
You probably already know, what happens when programming is more "real-time". It's fun again.
When you update code, SkyAlt will update running program
The original C code has 50K LOC(lines of code). The new C code has 24K LOC. I removed 32K LOC and added 5K LOC(mostly parser, interpreter).
The code in my new language is only 3K LOC. So I ported 32K lines of C to 3K of language which I made. Hmm, not bad.
The parser for new lang has 1K LOC and the interpreter has 4K LOC.
SkyAlt build 2.5(all in C lang) has 1.8MB. The build 3(C + new lang) has 1.0MB. So I removed cca 0.8MB of code. 3/4 million bytes of code with unknown bugs are gone now!
The Parser converts that 3K LOC to AST in 0.01 seconds. So when you start SkyAlt you can not feel it and when you update code it will take 1 second(it doesn't check If files have been changed every tick, but every second).
By the way, MSVC compiler compiles(release mode) the latest C code in 4 seconds and GCC finishes in 10 seconds.
You know, how predicting the future is hard. This is the case. I've built this language to solve one big problem: too many bugs. But the experience using the language is so good, that I think, I should provide it to SkyAlt user so they can extend SkyAlt very easily.
The most of current software in the big data market is fixed. Few ones offer "blocks", which you have to code as an extension or plugin.
Here's a simple idea. Powerful software with 2 basis features: creating and applying assets. The big idea here is to provide an environment where people(not just programmers) can easily create assets, which they can apply and shape their data. Since the language is for organizing and analyzing big data, creating assets has some boundaries but it's still way better than what offers other players in the field.
You start with an empty project, you use a connector(asset) to your favorite database and bring data in. You wanna filter out some records, you use the appropriate asset or you can make/edit one. You wanna see your data as a chart, pick the right asset. This goes beyond dragging and dropping blocks(plugins) on the dashboard. You will be able to extend any place in your project and share that "know-how" with others.
There are two very important parts, which have to work very well. (1) Quickly searching for an asset. (2) Easily connecting asset to your project. Especially 2nd one is hard to crack. I have a plan on how to do that on paper, but because connecting assets together can be complex, it's better If most of the users don't do that. The key thing is that users don't share only assets, but also how and where they are connected to each other. It's like when someone makes and publishes a library, but also example code for how to work with a library. But in SkyAlt, asset connections will be automated. This is also why it's very important to build and grow the community - more assets, more connections, which means easier to use.
This idea pushes SkyAlt to a different game. I was building fixed features that I thought users wants and now It's about creating a community of people who will build and share the features. This is one hell of the changeover.
If you wanna create domain-specific language, you need to have experiences in the specific domain. If you are building something which you know a little about, the only way how you get the experiences is that you spin out the language you know and write 5-15x more lines of code than what you would write in that "dream" language of yours. When you understand the domain, you are in a good position to make that "dream" language reality.
Just keep in mind, that DSL is mostly useful for long-term projects. For other people it will take some time to learn it, but long-term efficiency will be out of the roof!
My new language is not revolutionary or even evolutionary, it's just cleverly connected programming paradigms to solve 2 specific use-cases: Organizing and analyzing big data. It works very well.
Since I was able to port a very complex and large(for me) C codebase, I think that the language itself is pretty advanced and saved time in future development is great. If programming is art, this is the most creative programming, I've ever done. Now, It's time to give the same experience to everyone.
Milan Suk
1st June, 2020
