subtable creation performance for a few Ms

[heidricha]

Hello,

I found the subtable creation very slow.
I have to handle 500K devices, maximum 10 types of data (different sets of attributes) for each.

I created 10 stables, one per datatype.
Then I started using it (insert data), but it was very slow, especially when I inserted only one value per subtable (device)
I thought I had to create the subtables before starting the inserts, but the creation is equally slow: 1000 device, one datatype takes 44 sec. I need 500K and 10 types, so the projected time just to create the tables is 152 hours.

It cannot be valid, I guess, but what shall I do differently?

I use python and the "ws" module, my data is like:

        # create super tables
        self.execute("create table dt_1 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")
        self.execute("create table dt_2 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")
        self.execute("create table dt_3 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")
        self.execute("create table dt_4 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")
        self.execute("create table dt_5 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")
        self.execute("create table dt_6 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")
        self.execute("create table dt_7 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")
        self.execute("create table dt_8 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")
        self.execute("create table dt_9 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")
        self.execute("create table dt_10 (`ts` timestamp, `ct` int, `v` float, `s` int, `f` int, `dssi` int, `rri` int, `ui` int, `ri` int) tags (`serial` varchar(10))")

attribute sets are the same in this phase of the tests

The creation process is like (dt is the data type id):

    for meter in range(first_meter, first_meter+number_of_meters):
        serial=f"{meter:010g}"
        d.execute(f"create table dt_{dt}_{serial} using dt_{dt} tags ('{serial}')")

What shall I do differently? I cannot use the engine this way

[heidricha]

hmmm....Looks like using "taos" command line and generated SQL is a 1000 times faster.

unfortunately, I cannot execute many commands in a single execute call :(
ws and rest modules look like slow otherwise...

too bad the python "taos" module doesn't work as expected - but that's another story.