File system for GitHub

Related tags

File Handling hubfs
Overview



HUBFS · File System for GitHub

HUBFS is a read-only file system for GitHub and Git. Git repositories and their contents are represented as regular directories and files and are accessible by any application, without the application having any knowledge that it is really accessing a remote Git repository.

How to use

HUBFS is a command-line program with a usage documented below. On Windows there is also good desktop integration so that you can use HUBFS without the command line.

HUBFS supports both authenticated and non-authenticated access to repositories. When using HUBFS without authentication, only public repositories are available. When using HUBFS with authentication, both public and private repositories become available; an additional benefit is that the rate limiting that GitHub does for certain operations is relaxed.

In order to mount HUBFS issue the command hubfs MOUNTPOINT. For example, hubfs H: on Windows or hubfs mnt on macOS and Linux.

The first time you run HUBFS you will be prompted to authorize it with GitHub:

> ./hubfs H:
First, copy your one-time code: XXXX-XXXX
Then press [Enter] to continue in the web browser...

HUBFS will then open your system browser where you will be able to authorize it with GitHub. HUBFS will store the resulting authorization token in the system keyring (Windows Credential Manager, macOS Keychain, etc.). Subsequent runs of HUBFS will use the authorization token from the system keyring and you will not be required to re-authorize the application.

To unmount the file system simply use Ctrl-C. On macOS and Linux you may also be able to unmount using umount or fusermount -u.

File system representation

By default HUBFS presents the following file system hierarchy: / owner / repository / ref / path

  • Owner represents the owner of repositories under GitHub. It may be a user or organization. An owner is presented as a subdirectory of the root directory and contains repositories. However the root directory cannot be listed, because there are far too many owners to list.

  • Repository represents a repository owned by an owner. A repository is presented as a directory that contains refs.

  • Ref represents a git "ref". It may be a git branch, a git tag or even a commit hash. A ref is presented as a directory that contains repository content. However when listing a repository directory only branch refs are listed.

  • Path is a path to actual file content within the repository.

HUBFS interprets submodules as symlinks. These submodules can be followed if they point to other GitHub repositories (this currently works well only on Windows when using the -o rellinks option). General repository symlinks should work as well.

Windows integration

When you use the MSI installer under Windows there is better integration of HUBFS with the rest of the system:

  • There is a "Start Menu > HUBFS > Perform GitHub auth" shortcut that allows you to authorize HUBFS with GitHub without using the command line.

  • You can mount HUBFS drives using the Windows Explorer "Map Network Drive" functionality. To dismount use the "Disconnect Network Drive" functionality. (It is recommended to first authorize HUBFS with GitHub using the above mentioned shortcut.)

  • You can also mount HUBFS with the net use command. The command net use H: \\hubfs\github.com will mount HUBFS as drive H:. The command net use H: /delete will dismount the H: drive.

How to build

In order to build HUBFS run make from the project's root directory. On Windows you will have to run .\make. The build prerequisites for individual platforms are listed below:

How it works

HUBFS is a cross-platform file system written in Go. Under the hood it uses cgofuse over either WinFsp on Windows, macFUSE on macOS or libfuse on Linux. It also uses go-git for some git functionality.

HUBFS interfaces with GitHub using the REST API. The REST API is used to discover owners and repositories in the file system hierarchy, but is not used to access repository content. The REST API is rate limited (details).

HUBFS uses the git pack protocol to access repository content. This is the same protocol that git uses during operations like git clone. HUBFS uses some of the newer capabilities of the pack protocol that allow it to fetch content on demand. HUBFS does not have to see all of the repository history or download all of the repository content. It will only download the commits, trees and blobs necessary to back the directories and files that the user is interested in. Note that the git pack protocol is not rate limited.

HUBFS caches information in memory and on local disk to avoid the need to contact the servers too often.

Git pack protocol use

HUBFS uses the git pack protocol to fetch repository refs and objects. When HUBFS first connects to the Git server it fetches all of the server's advertised refs. HUBFS exposes these refs as subdirectories of a repository.

When accessing the content of a ref for the first time, the commit object pointed by the ref is fetched, then the tree object pointed by the commit is fetched. When fetching a tree HUBFS will also fetch all blobs directly referenced by the tree, this is required to compute proper stat data (esp. size) for files.

HUBFS fetches objects with a depth of 1 and a filter of tree:0. This ensures that the git server will only send objects whose hashes have been explicitly requested. This avoids sending extraneous information and speeds up communication with the server.

Security issues

  • Some applications insist on accessing files or directories from the root of the HUBFS file system (especially on Windows). Some of these accesses are disallowed by HUBFS (e.g. accesses for .git, HEAD or autorun.inf). Others are allowed which can create the possibility of a security issue for some applications, because anyone can create a username on GitHub that matches the name of a file that an application is looking for.

    • For example, consider an application that always accesses files under /billziss-gh/content. Because I own the billziss-gh account I could create a repository named content and inject file content into someone's process. While the /billziss-gh prefix is unlikely, other prefixes are far more likely.

    • To alleviate this problem one may start HUBFS with a "prefix" to restrict HUBFS usage only within a particular account or repository. For example, ./hubfs github.com/billziss-gh H: will access only my own repositories.

    • An alternative solution might involve adding an option to only allow access to a list of desired owners and their repositories.

Potential future improvements

  • The file system is currently read-only. This means that the common scenario of building inside the repository is not supported. In the future this problem may be alleviated either in this or another project.

  • The file system does not present a .git subdirectory. It may be worthwhile to present a virtual .git directory so that simple Git commands (like git status) would work.

  • Additional providers such as BitBucket, GitLab, etc.

License

This project is licensed under the MIT License.

Comments
  • Cannot access private repo owned by different user or organization

    Cannot access private repo owned by different user or organization

    I have a private repo on my account and orgs, shared with other users.

    When I use HUBFS, (on Windows, installed with .msi, authenticated with "Preform GitHub Auth") I can access my private repos without any issue. All features that I've tested work as expected.

    However, if I try to access a private repo owned by a different user or org, even if I'm the owner of the org, I cannot access them.

    They also do not show up in the top level user's (or org's) folder(s).

    If I try to go to the project's URL directly, I get a "Windows cannot access" error with code: 0x80004005 that looks identical to what you see when you access a nonexistent repo: image

    opened by cinderblock 6
  • [QUE] Can hubfs currently be used with a Gitlab Free on premise server?

    [QUE] Can hubfs currently be used with a Gitlab Free on premise server?

    Hello @billziss-gh,

    Beyond not having explicit Gitlab support yet, is there any possibility to use hubs with a Gitlab Free server hosted on premise?

    🙏Thanks for developing this useful software and making it accessible to the community.

    Regards, Claudio Salvio

    opened by claudio-salvio 5
  • refs other than heads(branches),tags,hashes

    refs other than heads(branches),tags,hashes

    Nice initiative 👏 I had a look at your source code and just wanted to confirm and highlight that your "ref" seems to only handle refs/heads/*, refs/tags/* and hashes. Except there can be other refs. git notes command and https://github.com/google/git-appraise store content under refs/notes, etc.

    opened by andreineculau 4
  • Changes to mounted repository are not kept

    Changes to mounted repository are not kept

    On:

    • Ubuntu 18.04

    I've followed the instructions in the README and managed to mount a Github private repository , authorizing my Ubuntu PC via the one-time code.

    I mount and then create / write top level files in the repo :

    $ hubfs ~/mnt/
    <... use one-time code and launch browser to authorize device aka PC ... >
    $ # in another terminal ... :
    $ cd ~/mnt/d-w-moore2/practice/master ; touch a ; vim README.md # make changes and save
    $ cd ~ ; fusermount -u mnt
    

    Then:

    • I do not notice Github in the browser showing any commits on masterto reflect the touched a file and README.md changes.
    • re-mounting and cd'ing into the directory again, I can confirm the changes have not been saved
    opened by d-w-moore2 2
  • Device flow is disabled

    Device flow is disabled

    When I try to run hubfs I get the following error

    {23:39}~ ➭ hubfs /mnt
    hubfs: client error: Device Flow must be explicitly enabled for this App (device_flow_disabled)
    

    I suspect this is due to the recent github change which means that GitHub now requires device flow to be enabled for apps rather than having it enabled by default and the hubfs app does not yet have this enabled.

    Update: a workaround in the meantime could be to use the -auth option to specify a different way of authenticating with GitHub. hubfs -auth optional /mnt, for example, doesn't give this error

    opened by Minion3665 1
  • Improve explanation of changes

    Improve explanation of changes

    This seems like an interesting product but it would be great if the readme explained more of how changes are handled. For example, if you change a file, is it then possible to commit that change?

    opened by Mellbourn 1
Releases(v1.0B1)
  • v1.0B1(Mar 8, 2022)

    HUBFS is a file system for GitHub and Git. Git repositories and their contents are represented as regular directories and files and are accessible by any application, without the application having any knowledge that it is really accessing a remote Git repository. The repositories are writable and allow editing files and running build operations.

    HUBFS

    Installation

    • Windows: Use either the MSI installer (if you want Explorer integration) or the ZIP file. You will also need to install the latest version of WinFsp.

    • Linux: Use the ZIP file, which should run on x64 glibc distros. You will also need FUSE.

    • macOS: Use the ZIP file, which contains a universal binary that should run on both x64 and arm64 systems. You will also need macFUSE.

    For more information see the project README.

    Source code(tar.gz)
    Source code(zip)
    hubfs-lnx-1.0.22067.zip(2.53 MB)
    hubfs-mac-1.0.22067.zip(2.63 MB)
    hubfs-win-1.0.22067.msi(3.26 MB)
    hubfs-win-1.0.22067.zip(2.52 MB)
  • v0.1B1(Apr 7, 2021)

    HUBFS · File System for GitHub

    This is the first release of HUBFS, the file system for GitHub. HUBFS is a cross-platform file system that has been tested on Windows, macOS and Linux. Currently only binaries for Windows are provided. Binaries for additional platforms will likely be provided in the near future.

    For more information on HUBFS see its README.

    Windows installation

    • Install the latest version of WinFsp.
    • Install HUBFS either via the MSI installer file or the compressed ZIP file. The MSI file provides machine-wide installation and better Windows integration (enabling features such as "Map Network Drive" and net use); however it requires Administrator privileges. The ZIP file simply provides the HUBFS executable without further system integration and the need for Administrator privileges; it can be used from the command line or scripts.
    Source code(tar.gz)
    Source code(zip)
    hubfs-win-0.1.21096.msi(3.34 MB)
    hubfs-win-0.1.21096.zip(2.45 MB)
Owner
Bill Zissimopoulos
Founder and owner at Navimatics LLC. Previously architect and manager at Netscape and Microsoft. Airplane, floatplane and helicopter pilot. Sailor.
Bill Zissimopoulos
Upgit - Upgit helps you simply upload any file to your Github repository and then get a raw URL for it

Upgit - Upgit helps you simply upload any file to your Github repository and then get a raw URL for it

null 321 Nov 23, 2022
Bigfile -- a file transfer system that supports http, rpc and ftp protocol https://bigfile.site

Bigfile ———— a file transfer system that supports http, rpc and ftp protocol 简体中文 ∙ English Bigfile is a file transfer system, supports http, ftp and

null 233 Oct 28, 2022
File system event notification library on steroids.

notify Filesystem event notification library on steroids. (under active development) Documentation godoc.org/github.com/rjeczalik/notify Installation

Rafal Jeczalik 773 Nov 30, 2022
Pluggable, extensible virtual file system for Go

vfs Package vfs provides a pluggable, extensible, and opinionated set of file system functionality for Go across a number of file system types such as

C2FO 204 Nov 21, 2022
Cross-platform file system notifications for Go.

File system notifications for Go fsnotify utilizes golang.org/x/sys rather than syscall from the standard library. Ensure you have the latest version

fsnotify 7.7k Dec 1, 2022
Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in o

dragonflyoss 6k Nov 26, 2022
Plik is a scalable & friendly temporary file upload system ( wetransfer like ) in golang.

Want to chat with us ? Telegram channel : https://t.me/plik_root_gg Plik Plik is a scalable & friendly temporary file upload system ( wetransfer like

root.gg 1.1k Nov 21, 2022
A virtual file system for small to medium sized datasets (MB or GB, not TB or PB). Like Docker, but for data.

AetherFS assists in the production, distribution, and replication of embedded databases and in-memory datasets. You can think of it like Docker, but f

mya 8 Feb 9, 2022
GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go

GeeseFS is a high-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go Overview GeeseFS allows you to mount an S3 bucket as a file sys

Yandex.Cloud 309 Nov 21, 2022
Encrypted File System in Go

Getting Started: Setup the environment: Install GoLang: $ sudo apt update $ sudo apt upgrade $ sudo apt install libssl-dev gcc pkg-config $ sudo apt

Lucky Verma 0 Apr 30, 2022
A rudimentary go program that allows you to mount a mongo database as a FUSE file system

This is a rudimentary go program that allows you to mount a mongo database as a

Jay Goel 1 Dec 29, 2021
Gokrazy mkfs: a program to create an ext4 file system on the gokrazy perm partition

gokrazy mkfs This program is intended to be run on gokrazy only, where it will c

null 4 Jun 13, 2022
A Go io/fs filesystem implementation for reading files in a Github gists.

GistFS GistFS is an io/fs implementation that enables to read files stored in a given Gist. Requirements This module depends on io/fs which is only av

Jean Hadrien Chabran 126 Oct 14, 2022
Small gh extension that suggests issues to work on in a given GitHub repository

gh contribute being a gh extension for finding issues to help with in a GitHub repository. This extension suggests an issue in a given repository to w

Nate Smith 26 Nov 2, 2022
Paste your GitHub Secrets to files

Paste-Secret Paste your GitHub Secrets in files Usage Inputs Required secrets : Secrets ise JSON object array. Holds filename, keys and values which w

Barış Can Yılmaz 8 Feb 25, 2022
A FileSystem Abstraction System for Go

A FileSystem Abstraction System for Go Overview Afero is a filesystem framework providing a simple, uniform and universal API interacting with any fil

Steve Francia 4.9k Nov 27, 2022
Experimental typesetting system

ETS Experimental typesetting system This software repository contains a Lua frontend for the typesetting library “Boxes and Glue” which is an algorith

speedata GmbH 4 Jan 30, 2022
Go-lang based sonos standup system

Overview This is an CLI tool that can handle timed standup playback on a sonos device. It allows you to add links to audio files that will be randomly

null 1 Nov 23, 2021