Browse Source

Fix some errors, update README, try to introduce SHA256 code verification.

Giovanni Mascellani 4 years ago
parent
commit
1a769a0775
10 changed files with 156 additions and 75 deletions
  1. 3 0
      .gitmodules
  2. 58 23
      README.md
  3. 1 1
      asmg/main.g
  4. 1 0
      contrib/sha-2
  5. 5 1
      diskfs/asmc_api.h
  6. 2 4
      diskfs/run_tcc.c
  7. 71 12
      diskfs/run_tcc_ipxe.c
  8. 1 0
      diskfs/sha-2
  9. 2 2
      http/continue.c
  10. 12 32
      http/continue2.c

+ 3 - 0
.gitmodules

@@ -19,3 +19,6 @@
 [submodule "contrib/single_cream"]
 	path = contrib/single_cream
 	url = https://github.com/rain-1/single_cream.git
+[submodule "contrib/sha-2"]
+	path = contrib/sha-2
+	url = https://github.com/amosnier/sha-2.git

+ 58 - 23
README.md

@@ -2,14 +2,13 @@
 
 `asmc` is an extremely minimal operating system, whose binary seed
 (the compiled machine code that is fed to the CPU when the system
-boots) is very small (around 15 KiB, maybe could be further shrunk in
-the future). Such compiled code is enough to contain a minimal IO
-environment and compiler, which is used to compile a more powerful
-environment and compiler, further used to compile even more powerful
-things, until a fully running system is bootstrapped. In this way
-nearly any code running in your computer is compiled during the boot
-procedure, except the the initial seed that ideally is kept as small
-as possible.
+boots) is very small (around 6 KiB). Such compiled code is enough to
+contain a minimal IO environment and compiler, which is used to
+compile a more powerful environment and compiler, further used to
+compile even more powerful things, until a fully running system is
+bootstrapped. In this way nearly any code running in your computer is
+compiled during the boot procedure, except the the initial seed that
+ideally is kept as small as possible.
 
 This at least is the plan; from the moment we are not yet at the point
 where we manage to compile a real system environment. However, work is
@@ -23,8 +22,49 @@ a C compiler. In the end a different path was devised: the initial
 seed is written in Assembly and embeds a G compiler (where G is a
 custom language, sitting something between Assembly and C, conceived
 to be very easy to compile); the G compiler is then use to produce a C
-compiler. Assembly is never directly used in this chain, although of
-course continuously behind the curtains.
+compiler. Assembly is never directly used in this chain, although it
+is of course continuously behind the curtains.
+
+`asmc` is currently able to:
+
+ * boot from a minimal seed of around 6 KiB;
+
+ * compile and execute source code written in the G language (see
+   below);
+
+ * initialize a minimal environment, including a terminal (writing to
+   serial port and to monitor), dynamic memory allocation, simple data
+   structures, disk access and a simple virtual file system;
+
+ * compile a basic assembler and a basic C compiler, both of which are
+   written in G;
+
+ * use them to compile a [lightly patched
+   version](https://gitlab.com/giomasce/tinycc/tree/softfloat) of
+   [tinycc](http://www.tinycc.org/), using a custom basic C standard
+   library;
+
+ * use tinycc to compile another copy of itself (in line of principle
+   that can be repeated as long as you want);
+
+ * use the second tinycc to compile a [patched and customized
+   version](https://gitlab.com/giomasce/ipxe/) of
+   [iPXE](http://ipxe.org/);
+
+ * use iPXE to initialize the network card and be ready to download
+   more source code from the network.
+
+Hopefully `asmc` will eventually be able to:
+
+ * download the Linux source code, [patch
+   it](https://gitlab.com/giomasce/linux/tree/tcc2) and compile via a
+   custom build script;
+
+ * compile a minimal userspace environment, which includes at least a
+   copy of tinycc;
+
+ * boot Linux with this userspace, and be ready to compile other
+   source code to continue bootstrapping.
 
 ## Enough talking, show me the deal!
 
@@ -42,24 +82,19 @@ the submodules as well:
 
 Then just call `make` in the toplevel directory of the repository. A
 subdirectory named `build` will be created, with all compilation
-artifacts inside it. In particular `build/boot_asmg.x86.qcow2` is a
-bootable disk image, which you can run with QEMU:
+artifacts inside it. Before running the virtual machine you have to
+start the web server that serves code for the later stages of `asmc`:
+open another terminal, enter the `http` directory and run
+
+    python3 -m http.server 8080
 
-    qemu-system-i386 -m 256M -hda build/boot_asmg.x86.qcow2 -serial stdio -device isa-debug-exit -display none
+Then:
+
+    qemu-system-i386 -m 256M -hda build/boot_asmg.x86.qcow2 -serial stdio -display none
 
 (if your host system supports it, you can add `-enable-kvm` to benefit
 of KVM acceleration)
 
-Unless I have broken something, this should run a little operating
-system that compiles a little C compiler, and later uses such compiler
-to compile from sources a [patched
-version](https://gitlab.com/giomasce/tinycc/tree/softfloat) of
-[tinycc](http://www.tinycc.org/), which is then used to compile a
-little test C program. In the future, tinycc will be used to continue
-the chain and build a Linux kernel and GNU userspace, so that you will
-actually have a complete operating system entirely compiled from
-scratches at computer boot!
-
 WARNING! ATTENTION! Remember that you are giving full control over you
 hardware to an experimental program whose author is not an expert
 operating system programmer: it could have bugs and overwrite your

+ 1 - 1
asmg/main.g

@@ -20,7 +20,7 @@ const TEST_MAP TEST_ALL
 const TEST_INT64 TEST_ALL
 const TEST_C TEST_ALL
 
-const RUN_MM0 1
+const RUN_MM0 0
 const RUN_ASM 0
 const RUN_FASM 0
 const RUN_C 0

+ 1 - 0
contrib/sha-2

@@ -0,0 +1 @@
+Subproject commit ff76937294694ec347819d81a1f580187fb34246

+ 5 - 1
diskfs/asmc_api.h

@@ -2,5 +2,9 @@
 #include "asmc.h"
 #include "ipxe_asmc_structs.h"
 
-downloaded_file *download_file(const char *url, const char *dest_file);
+#include "sha-2/sha-256.h"
+
+void download_file(const char *url, const char *dest_file);
 void *compile_file(const char *code, const char *symbol, void **buffer);
+void *read_whole_file(FILE *fin, size_t *size);
+void compute_file_sha256(const char *filename, uint8_t hash[32]);

+ 2 - 4
diskfs/run_tcc.c

@@ -27,17 +27,15 @@ int recursively_compile() {
     includes[6] = ASMC_PREFIX "/tinycc/softfloat/8086";
     const char *files[1];
 
+    printf("Compiling tinycc recursively... (level %d)\n", LEVEL);
 #if LEVEL == 1 && defined(RUN_IPXE)
 #define ADD_TCC_SYMBOLS
     files[0] = ASMC_PREFIX "/run_tcc_ipxe.c";
-    printf("I will now try to compile and execute tinycc, diverting to iPXE! (level %d)\n", LEVEL);
 #elif LEVEL == 1 && defined(RUN_SINGLE_CREAM)
 #define ADD_TCC_SYMBOLS
     files[0] = ASMC_PREFIX "/run_tcc_sc.c";
-    printf("I will now try to compile and execute tinycc, diverting to single_cream! (level %d)\n", LEVEL);
 #else
     files[0] = ASMC_PREFIX "/run_tcc.c";
-    printf("I will now try to compile and execute tinycc recursively! (level %d)\n", LEVEL);
 #endif
 
     TCCState *state = tcc_new();
@@ -82,7 +80,7 @@ int recursively_compile() {
     }
     printf("Entering tinycc...\n");
     int ret = start();
-    printf("The program returned %d!\n", ret);
+    printf("Tinycc returned %d!\n", ret);
 
     tcc_delete(state);
     return 0;

+ 71 - 12
diskfs/run_tcc_ipxe.c

@@ -157,6 +157,37 @@ const char *sources[][2] = {
 #include "ipxe_handover.h"
 #include "asmc_api.h"
 
+#include "sha-2/sha-256.c"
+
+void *read_whole_file(FILE *fin, size_t *size) {
+    size_t buf_size = 16;
+    char *buf = malloc(buf_size);
+    size_t buf_pos = 0;
+    while (1) {
+        size_t res = fread(buf + buf_pos, 1, buf_size - buf_pos, fin);
+        buf_pos += res;
+        if (buf_pos < buf_size) {
+            break;
+        } else {
+            buf_size *= 2;
+            buf = realloc(buf, buf_size);
+        }
+    }
+    if (size) {
+        *size = buf_pos;
+    }
+    return buf;
+}
+
+void compute_file_sha256(const char *filename, uint8_t hash[32]) {
+    FILE *fin = fopen(filename, "rb");
+    size_t size;
+    void *buf = read_whole_file(fin, &size);
+    fclose(fin);
+    calc_sha_256(hash, buf, size);
+    free(buf);
+}
+
 int table_comp(void *t1, void *t2) {
     table_sect *s1 = t1;
     table_sect *s2 = t2;
@@ -260,18 +291,36 @@ void *pop_from_ipxe() {
 
 coro_t *coro_ipxe;
 
-downloaded_file *download_file(const char *url, const char *dest_file) {
+void download_file(const char *url, const char *dest_file) {
     push_to_ipxe(strdup("download"));
     push_to_ipxe(strdup(url));
     coro_enter(coro_ipxe);
     downloaded_file *df = pop_from_ipxe();
     assert(df->size > 0);
-    /*printf("Received document of %d bytes!\n", df->size);
-    printf("---\n");
-    for (size_t i = 0; i < df->size; i++) {
-        putchar(df->data[i]);
+    FILE *fout = fopen(dest_file, "wb");
+    fwrite(df->data, 1, df->size, fout);
+    fclose(fout);
+    free(df->data);
+    free(df);
+}
+
+void download_check_file(const char *url, const char *dest_file, const uint8_t hash[32]) {
+    push_to_ipxe(strdup("download"));
+    push_to_ipxe(strdup(url));
+    coro_enter(coro_ipxe);
+    downloaded_file *df = pop_from_ipxe();
+    assert(df->size > 0);
+    uint8_t computed_hash[32];
+    calc_sha_256(computed_hash, df->data, df->size);
+    printf("Computed hash:");
+    size_t i;
+    for (i = 0; i < 32; i++) {
+        printf(" %02x", computed_hash[i]);
+    }
+    printf("\n");
+    for (i = 0; i < 32; i++) {
+        assert(computed_hash[i] == hash[i]);
     }
-    printf("---\n");*/
     FILE *fout = fopen(dest_file, "wb");
     fwrite(df->data, 1, df->size, fout);
     fclose(fout);
@@ -307,29 +356,32 @@ void *compile_file(const char *filename, const char *symbol, void **buffer) {
     tcc_add_symbol(state, "tcc_get_symbol", tcc_get_symbol);
     tcc_add_symbol(state, "download_file", download_file);
     tcc_add_symbol(state, "compile_file", compile_file);
+    tcc_add_symbol(state, "calc_sha_256", calc_sha_256);
+    tcc_add_symbol(state, "read_whole_file", read_whole_file);
+    tcc_add_symbol(state, "compute_file_sha256", compute_file_sha256);
     int res;
     for (int i = 0; i < sizeof(includes2) / sizeof(includes2[0]); i++) {
         res = tcc_add_include_path(state, includes2[i]);
         if (res) {
             printf("tcc_add_include_path() failed...\n");
-            return 1;
+            return 0;
         }
     }
     res = tcc_add_file(state, filename);
     if (res) {
         printf("tcc_add_file() failed...\n");
-        return 1;
+        return 0;
     }
     int size = tcc_relocate(state, NULL);
     if (size < 0) {
         printf("tcc_relocate() failed...\n");
-        return 1;
+        return 0;
     }
     *buffer = malloc(size);
     size = tcc_relocate(state, *buffer);
     if (size < 0) {
         printf("second tcc_relocate() failed...\n");
-        return 1;
+        return 0;
     }
     void *symb = tcc_get_symbol(state, symbol);
     tcc_delete(state);
@@ -416,10 +468,17 @@ int main(int argc, char *argv[]) {
 
     // Download the continuation program
     const char *url = "http://10.0.2.2:8080/continue.c";
-    printf("Request to download %s\n", url);
+    printf("Request to download %s...\n", url);
+    const uint32_t hash[32] = {0xEB, 0x42, 0xD9, 0x44, 0xC2, 0x2E, 0x46, 0x4F, 0x39, 0xAF, 0x75, 0x31, 0xED, 0x74, 0x99, 0xA1,
+                               0x1D, 0xC5, 0xDC, 0x6E, 0x34, 0x77, 0x3C, 0x35, 0xFE, 0x14, 0x1B, 0xAC, 0x5D, 0x53, 0x3D, 0x5D};
     download_file(url, "/ram/continue.c");
+    //download_check_file(url, "/ram/continue.c", hash);
     void *cont_buf;
     int (*cont_symb)(int, char*[]) = compile_file("/ram/continue.c", "_start", &cont_buf);
+    if (!cont_symb) {
+        printf("Compilation failed...\n");
+        return 1;
+    }
     char *cont_argv[] = {"_main"};
     cont_symb(sizeof(cont_argv) / sizeof(cont_argv[0]), cont_argv);
     free(cont_buf);
@@ -430,7 +489,7 @@ int main(int argc, char *argv[]) {
 
     //res = main_symb(&ih);
 
-    printf("Returning from iPXE!\n");
+    printf("iPXE exited!\n");
     coro_destroy(coro_ipxe);
     free_handover(&ih);
     free(ipxe_buf);

+ 1 - 0
diskfs/sha-2

@@ -0,0 +1 @@
+../contrib/sha-2

+ 2 - 2
http/continue.c

@@ -2,8 +2,8 @@
 #include <stdio.h>
 #include <asmc_api.h>
 
-int main() {
-    printf("Hello, continuing world!\n");
+int main(int argc, char *argv[]) {
+    printf("Hello world from continue.c!\n");
     download_file("http://10.0.2.2:8080/continue2.c", "/ram/continue2.c");
     void *cont2_buf;
     int (*cont2_symb)(int, char*[]) = compile_file("/ram/continue2.c", "_start", &cont2_buf);

+ 12 - 32
http/continue2.c

@@ -468,33 +468,13 @@ void prepare_real_mode_gdt() {
     gdt[3].base_high = 0x00;
 }
 
-void *read_all_file(FILE *fin, size_t **size) {
-    size_t buf_size = 16;
-    char *buf = malloc(buf_size);
-    size_t buf_pos = 0;
-    while (1) {
-        size_t res = fread(buf + buf_pos, 1, buf_size - buf_pos, fin);
-        buf_pos += res;
-        if (buf_pos < buf_size) {
-            break;
-        } else {
-            buf_size *= 2;
-            buf = realloc(buf, buf_size);
-        }
-    }
-    if (size) {
-        *size = buf_pos;
-    }
-    return buf;
-}
-
 //struct boot_params boot_params __attribute__((aligned(16)));
 
 void prepare_setup_header(struct setup_header *hdr, void *initrd, size_t initrd_size) {
     hdr->vid_mode = 0xffff;
     hdr->type_of_loader = 0xff;
     hdr->loadflags = 0x81;
-    hdr->ramdisk_image = initrd;
+    hdr->ramdisk_image = (uint32_t) initrd;
     hdr->ramdisk_size = initrd_size;
     hdr->heap_end_ptr = 0xde00;
     hdr->cmd_line_ptr = 0x1e000;
@@ -531,12 +511,12 @@ void load_linux32() {
     assert(real_mode->hdr.boot_flag == 0xaa55);
     assert(real_mode->hdr.header == 0x53726448);
     size_t prot_mode_size;
-    void *prot_mode = read_all_file(fin, &prot_mode_size);
+    void *prot_mode = read_whole_file(fin, &prot_mode_size);
     printf("Protected mode code/data is 0x%x bytes long\n", prot_mode_size);
     fclose(fin);
     fin = fopen("/ram/cont/initrd.img", "rb");
     size_t initrd_size;
-    void *initrd = read_all_file(fin, &initrd_size);
+    void *initrd = read_whole_file(fin, &initrd_size);
     printf("Initrd is 0x%x bytes long\n", initrd_size);
     fclose(fin);
     // Align initrd
@@ -548,7 +528,7 @@ void load_linux32() {
     char *cmdline = "verbose";
 
     // Here we really smash our operating system!
-    memset(0x10000, 0, 0x10000);
+    memset((void*) 0x10000, 0, 0x10000);
     memcpy((void*) 0x10000, real_mode, real_mode_size);
     memcpy((void*) 0x1e000, cmdline, strlen(cmdline));
     //memcpy((void*) 0x100000, prot_mode, prot_mode_size);
@@ -591,21 +571,21 @@ void load_linux() {
     assert(real_mode->hdr.boot_flag == 0xaa55);
     assert(real_mode->hdr.header == 0x53726448);
     size_t prot_mode_size;
-    void *prot_mode = read_all_file(fin, &prot_mode_size);
+    void *prot_mode = read_whole_file(fin, &prot_mode_size);
     printf("Protected mode code/data is 0x%x bytes long\n", prot_mode_size);
     fclose(fin);
     prepare_setup_header(&real_mode->hdr, 0, 0);
     //load_linux_gdt();
     fin = fopen("/ram/cont/realmode.bin", "rb");
     size_t trampoline_size;
-    void *trampoline = read_all_file(fin, &trampoline_size);
+    void *trampoline = read_whole_file(fin, &trampoline_size);
     printf("Trampoline is 0x%x bytes long\n", trampoline_size);
     fclose(fin);
     prepare_real_mode_gdt();
     char *cmdline = "verbose";
 
     // Here we really smash our operating system!
-    memset(0x10000, 0, 0x10000);
+    memset((void*) 0x10000, 0, 0x10000);
     memcpy((void*) 0x500, trampoline, trampoline_size);
     memcpy((void*) 0x10000, real_mode, real_mode_size);
     memcpy((void*) 0x1e000, cmdline, strlen(cmdline));
@@ -622,25 +602,25 @@ void load_mlb() {
     download_file("http://10.0.2.2:8080/example", "/ram/cont/bootloader");
     FILE *fin = fopen("/ram/cont/realmode.bin", "rb");
     size_t trampoline_size;
-    void *trampoline = read_all_file(fin, &trampoline_size);
+    void *trampoline = read_whole_file(fin, &trampoline_size);
     printf("Trampoline is 0x%x bytes long\n", trampoline_size);
     fclose(fin);
     prepare_real_mode_gdt();
     fin = fopen("/ram/cont/bootloader", "rb");
     size_t bootloader_size;
-    void *bootloader = read_all_file(fin, &bootloader_size);
+    void *bootloader = read_whole_file(fin, &bootloader_size);
     printf("Bootloader is 0x%x bytes long\n", bootloader_size);
     fclose(fin);
     prepare_real_mode_gdt();
 
-    memset(0x10000, 0, 0x10000);
+    memset((void*) 0x10000, 0, 0x10000);
     memcpy((void*) 0x500, trampoline, trampoline_size);
     memcpy((void*) 0x7c00, bootloader, bootloader_size);
     ((void (*)()) 0x500)();
 }
 
-int main() {
-    printf("Hello, continuing second world!\n");
+int main(int argc, char *argv[]) {
+    printf("Hello world from continue2.c!\n");
     //load_linux32();
     //load_linux();
     //load_mlb();