Want to discuss an idea or a bug? Open up a new issue and we can talk about it. Pull requests are welcome.
Want to discuss an idea or a bug? Open up a new issue and we can talk about it. Pull requests are welcome.
Hope all is well.... I am really fascinated with your project and wider implications of this.
I am not sure if this issue / question shoudl be here or in the FS repo but here goes....
One thing i am working on is the relationship between retained mode GUI's and data caches. A retained mode gui can be conceptually designed like a data cache system that we all know and love. A gui has a template that is computed on to produce some markup. In a retained mode gui system you want to know if the template has changed. It woudl be a signal at the control plane level (if you get my drift), so you can do optimistic re rendering of that gui component instance, which is sort of like eager caching. Hope that this is making sense :)
Some background might also help... I know this is pretty out there... The DOM is a browser is doing the retained mode diffing for you. It knows when you changed the HTML in the inspector, and magically updates the HTML Screen for you. But when you working at the WASM level and rendering with WASM to a Webgl canvas, you need to do this yourself..
So now i get to my question... The FS implementation is super cool because it can work with a WASM, local or remote context, but i was wondering if it supports etags style caching or other meta data semantics to tell the caller that you need to do a cache miss and refetch because your local data is old buddy !!
Golang Http FS has etags: https://go.googlesource.com/go/+/go1.8.3/src/net/http/fs.go; see line 119. So it can easily act as a cache to a DB called over http for example.
The reason i bring this up is because you also want to do the same thing inside the WASM environment for many use cases. Let say in the WASM environment you have a FS ( that is indexeddb under the hood) that is sourced from S3. When the S3 source changes, i presume there is some sort of etag header to tell you. So you want when in the WASM environment there is a call to the WASM FS, it should realise that the local FS is now "dirty" or "invalidated" and so it should automatically get the data from S3, update the local file, and return that to the caller.
So i am wondering if this is currently supported.
It will be really interesting to see how this also related to the Web Worker and Service Worker stuff too Service Worker.
It will be really interesting to build quasi CDN like architectures where you can have caches at the WASM, local and remote level like a CDN style architecture. An thew also cross mix. Markup templates are dependent on data which can be dependent on the caache and its "turtles all the way down"..... I like the simplicity of this.
You also get into a interesting situation with Design time versus Runtime also. I use the work real time because the IDE is running in a browser and doing compilation as you change code. At non real time, you working off already compiled golang. At real time Design time, you want your goimport to be always latest and when someone else PR#s some code that you import you want to know NOW and see it and adapt. you want to get broken by them ... At Real time Runtime, you want to use a static version, and not break.
The same does for data. YOu have static data ( json, binaries, etc) and you have dynamic data. At Real time Design time, you want everything to be dynamic. Don't use any caching anywhere. At real time Runtime, you want everything that is static to never update, and everything that's dynamic to use the cache mechanism.
Hello and thank you for making this! For some reasons I cannot understand, the build process seems to be very slow. I am waiting for more than 5 minutes and still it didn't finish. I wonder if it is browser related;I am using Firefox un GNU/Linux 89 on a 64x machine.
This definitely is a fascinating project. The docker ceo apparently said that if WASM existed there would have been no need for docker. He was referring to sand boxing.
Did you hit many bugs when attempting compilation using tinygo. I think they did a release recently or it might still be on a branch . Tinygo is efficient enough that you could run on mobile or have a good chance.
I have a interesting idea. Your sage accounting software uses a bucket file system dB and a git based vcs to presumerable allow sync of the underlying files . Would be interesting test case if these two work running on top of the wasm indexdb / FS. It’s a decent test case I feel.
Would badger dB in theory work ?
Go routines A and B call emit(). Both see that visited == false under read lock. A acquires write lock and processes. B acquired right lock and double processes, even though A marked the node as visited. The fix is to check if visited again after write lock is acquired
Seeing the following error, see linked screenshot below:
Terminal text here for convenience also:
$ go version go version go1.16.6 js/wasm $ go mod init playground go: creating new go.mod: module playground go: to add module requirements and sums: go mod tidy $ go mod tidy $ go build -v . main.go:4:2: package fmt is not in GOROOT (/usr/local/go/src/fmt) package playground: cannot find package exit status 1
Any ideas what I might be doing wrong here? Thanks!
Switches to hackpadfs for filesystem setup, tests, and implementations. Parts of its original ideas were extracted from go-wasm, but it evolved into a totally new implementation.
Hackpadfs has proper tests, interface design, and documentation. All of which should help improve stability, isolation, and performance of go-wasm in the long run.
Further, the indexeddb adapter is much improved in its new home go-indexeddb.
Add vet and cover
go tool support.
Also add a
jsdownload builtin command for the shell. This makes it possible to generate coverage files for Wasm-only tests and then parse them on your own machine.
# in Go Wasm go test ./... -coverprofile=cover.out jsdownload cover.out # locally go tool cover -html cover.out
Perhaps eventually coverage can be fully integrated into the editor itself.
When one go beyond the editor viewport, the scrollbar overlaps the original one (which doesn't seem to have any utility). So, in order to avoid this situation, the 'original' vertical scrollbar has been remove.
Lowers memory usage for running binaries like
compile, and user programs.
Memory decreased by up to 56% in the best case and 47% in the worst case. The heap footprint should now remain below 250 MB in the worst case, down from 540 MB. On a second boot (not installing the Go toolchain), memory usage is reduced even further to below 180 MB. Execution times shortened a bit, but probably within margin of error.
See below for single samples of performance changes. These are super rough, so take them with a grain of salt.
I'd summarize the commits, but they were enough changes it'd probably be best to read through the commit messages themselves.
dompackage for clearer interop and compile safety in the editor
/home/mein the shell
The net of all of the below changes is: Significantly reduced memory usage and persistent files. Typical memory usage has dropped from ~750 MB to ~250 MB. This is a big leap architecturally, but we still have room to improve going forward. 🎉
Since the outrageous memory usage in the current release causes crashes on typical devices, we needed to store files somewhere other than RAM. In switching to a non-RAM storage medium, naturally go-wasm slowed down a bit. This speed difference can be further reduced in future PRs.
As a bonus, Safari now almost works as well as Chrome and Firefox. It's still rather unhappy with the memory usage, though. Pro tip: Safari basically hangs if you run
WebAssembly.compile around 100 times on the same data. Best to avoid that.
tarfsfor execution speed. In particular, utilizes "batched parallelism" to queue up large numbers of file operations and completing them all in one pass on the FS end.
tarfscustom file storage and replaced with an
afero.Fsinterface. Enabled easy swap-in of IDB FS.
blob.Blobabstraction, to significantly reduce conversion costs between
chdiron a strictly asynchronous valid directory file system check. It works, but it ain't great.
~/.history. Persisted with home dir IDB FS.
go mod tidyto first boot.
PATHenvironment variable propagation.
decode()is surprisingly slow, relative to everything else.
shmore robust, with
touchsupport, plus a new long format
Thanks to @guest271314 for the helpful nudge #4 ❤️
It would be cool to have a compiler as a service running on a server that the HackPad IDE can interact with.
the reason is because I am interested in making plugins . You write it in golang and compile with Tinygo . The Compiler As A Service ( CAAS ) does that and then you can “ see it “ in the IDE.
I saw the Web Worker Par in HackPadFS and it got me thinking. I presume that’s so that the FS in running isolated in its own isolated area outside of the main GUI and so hence hardening the system from crashes.
so my rational is to do it for all code. The developers code that they are developing in the IDE is compiled and exposed as a Web Worker.
The same can then be done with Wazero so that you can run the wasm on the server also. Why think about this ? Because you end up with reconfigurable ( is that a real word ? ) computing, and so depending on the nature of the code can run it in a Browser or on a Server ( with Wazero ).
The way WASI and WASM exposes the exports is different . But it’s possible to shim that by writing a compile time wrapping that walks the AST and code generates the equivalents needs for it to run in the browser too. If we write the exports out as a JSON file then an IDE can “ discover “ the exports and elevate it in the IDE GUI.
@codefromthecrypt as I figure this is interesting to you .
I probably should have made this a discussion.
My request is to know what are the blockers. is my codegen concept the way to go for at least the golang community. I read about WIX but the more I delve into it I get the impression that there is no standard yet ( or ever will be ) for WASI and WASM being able to be single sourced and run in both environment easily . Hence sone compile time and runtime wrapper ingredients seems needed.
The isolation aspects of WebWorker and Service Workers can possibly mapped to similar topology primitives on the Server works. A WebWorker is easy because it’s the same as a Fly.io Machine or a Wazero server hosting WASI. the Service Worker equivalent would be a gateway / Proxy / Caching type of thing similar to Caddy with Caddy Modules but not sure . Service workers have logic , caching and procuring capabilities.
Marked RFC until these are merged:
Hey, This is super cool. I have wanted for quite some time to build tutorials for the Go language using wasm (in browser) - you have probably jump started my work here by a mile. The first step is to make a super interactive platform and a 'framework' for the tutorials so its fast and easy to add new ones.
left hand side panel where the instructions for this step/tutorial are
I looked through the code and see you are designing/rendering the UI in Go which was amazing/surprising. Two questions really - one is do you have a link to tutorial in how you are doing this, and secondly, could this be swapped out for HTML/CSS/JS or React or something easily? My guess is WASM means better to do all in Go, so question one is more the focus - i.e can you point me to anything to learn how to adjust the UI?
pre written tests that can check the results are correct - I don't want to use regex parsing stdout or something, I want real Go tests to run against functions (students add their function call to the test harness so they can call functions whatever they want - part of learning is the tests.
This sounds possible based on your "OS" - Would be cool if tests can run against code?
I can use mockers (smocker) to mock out external services in tutorials I can run other Go apps/binaries that expose ports or something to interact with (e.g an API running locally)
I suspect these last two are closely related and its either "thats easy" or "thats hard" as the OS isn't really "real".....
Anyway would love just to hear your thoughts on this usecase for this platform. Thanks
Had some time to jump into this rabbit hole again and playing around with File System explorer module, so you get a tree.
On origin/feature/web-workers-2 branch
This is my simple make script that wraps your make script..
"make serve" does not work for me. Any ideas ?
build-init: # installs special go. cd $(REPO_NAME) && $(MAKE) go build: # builds the many wasm and then the server npm stuff. cd $(REPO_NAME) && $(MAKE) build serve: cd $(REPO_NAME) && $(MAKE) serve
This is currently a huge refactor and a work in progress. Hopefully will suss out the major bugs soon 😅