Attention Data Enthusiasts!🌟
Introducing Elusion Library, the brand-new data manipulation PowerHouse that combines the user-friendly syntax similar to Python libraries: Pandas and Polars, with blazing efficiency of Rust. Elusion seamlessly handles in-memory data formats like CSV, Parquet, JSON, and Delta Tables, ensuring versatile and high-performance data processing for any project.
Transcript
1 00:00:02.020 --> 00:00:26.390 Gabor Szabo: Hello, and welcome to the Codeme Event Channel. My name is Gabor Sabo. I usually teach rust and python at companies, and I also help introducing testing to companies and continuous integration and trying to help improving their development environment in in general. In addition, I think it's sharing
2 00:00:26.670 --> 00:00:52.369 Gabor Szabo: information, sharing knowledge is a very important thing in our in our community. And so we have this channel in the Youtube channel called me events. And these meetings that I'm organizing and inviting people from around the world to share their knowledge about rust. So if you are interested, 1st of all, please like the video and join and follow the Youtube channel. If you are watching this in Youtube.
3 00:00:52.960 --> 00:01:17.310 Gabor Szabo: also join or below the video, you're going to find a link to where you can join or group, and I'm really interested in in hearing suggestions for new other presentations about rust. And now I would like to thank you for for coming and and for joining us and giving this presentation. So
4 00:01:17.430 --> 00:01:24.410 Gabor Szabo: please, yeah, please introduce your yeah.
5 00:01:24.410 --> 00:01:25.539 Borivoj: Okay. Okay.
6 00:01:25.540 --> 00:01:26.889 Gabor Szabo: Go ahead, go ahead, sir. Just.
7 00:01:26.890 --> 00:01:30.230 Borivoj: It's hard to. It's hard to remember and to pronounce my name.
8 00:01:30.230 --> 00:01:40.030 Gabor Szabo: Yeah, yeah, yeah, I'm going to pronounce your name. I'm not sure how to pronounce it. My name so
9 00:01:40.030 --> 00:01:45.160 Gabor Szabo: difficult for pronunciation, for non-hungarian speakers. Okay, so.
10 00:01:45.160 --> 00:01:45.780 Borivoj: Yeah. So
11 00:01:46.530 --> 00:01:54.640 Borivoj: no problem. So my name is. But everybody calls me Bora, like in Ireland, Bora, Bora, this is the easiest way to remember.
12 00:01:55.420 --> 00:02:13.800 Borivoj: yeah, when I was younger, I was working on cruise ships. So all Americans, it's for them. It's very easy to remember. Bora Bora. Bora. Yeah. So Bora is simple to remember and and to pronounce, yeah. So I'm coming from Serbia. I'm 37 years old in a couple of days. Have 2 beautiful kids, 10 and 12.
13 00:02:14.280 --> 00:02:23.479 Borivoj: I'm a data engineering expert group currently working for on different projects, working mostly with with like American clients, healthcare companies.
14 00:02:23.620 --> 00:02:37.639 Borivoj: And I want to shout out to my colleagues that are working now without me, because usually now it's working time for us, you know. Second shift from 2 to 10. We usually work. So yeah.
15 00:02:37.740 --> 00:02:40.160 Borivoj: to cover this, you know, difference time difference.
16 00:02:40.840 --> 00:02:52.800 Borivoj: Anyways, yeah, I work as a data engineer, mostly working in fabric environment. Microsoft. And yeah, you're gonna hear a little bit more about this in in in the talk. But this is just a quick introduction of myself.
17 00:02:53.650 --> 00:02:55.550 Gabor Szabo: Go, go ahead, share your slide and.
18 00:02:55.810 --> 00:02:57.729 Borivoj: Yes. Okay, so let me just.
19 00:02:57.730 --> 00:03:17.079 Gabor Szabo: Just one more thing. Anyone in the group in in who is around now the live presentation feel free to ask questions. Obviously, you'll be in the video or or ask questions in the chat. If you prefer that way, and I'll when I see it, I'll read it, read out the questions.
20 00:03:17.480 --> 00:03:40.770 Borivoj: Okay, so you can, you can see the slide. Okay? So very good. So illusion, data frame library for everybody. So why, I'm usually saying like this, because my whole idea was to create something that will be easy to understand and easy to use by everybody in data world. So everybody who knows sequel. Everybody who knows data frame different data frame libraries.
21 00:03:41.090 --> 00:03:45.970 Borivoj: My goal was from the 1st day on when they come in to to understand. Okay.
22 00:03:46.130 --> 00:03:51.470 Borivoj: So as you can see below, I just want to mention one more time. Shout out to my guys at Expert group.
23 00:03:51.640 --> 00:04:04.309 Borivoj: I'm gonna see you later, after this talk. So what we're gonna talk about this today about a little bit about me. I'm gonna try to skip this fast. What is the motivation and what motivates me to to create this library? Why, another
24 00:04:04.590 --> 00:04:09.429 Borivoj: data frame library? Right? We. We have such a great libraries already in python world.
25 00:04:09.960 --> 00:04:27.190 Borivoj: And we're going to talk a little bit about data fusion, because we cannot talk about illusion without data fusion, because I'm heavily leaning on this SQL engine of data fusion. So I'm going to talk about illusion a little bit. Then we're gonna start covering a little bit of
26 00:04:27.680 --> 00:04:33.380 Borivoj: syntax, and why I decide to do and how I designed the syntax and why I did what I did.
27 00:04:33.740 --> 00:04:39.330 Borivoj: Okay, so the whole thing is, the whole idea is to promote the the library, not myself. So I gonna just
28 00:04:39.610 --> 00:04:40.709 Borivoj: quickly skip
29 00:04:41.020 --> 00:05:03.529 Borivoj: this part. So I finished music school. I think this is very important to have any sort of arts in your in your background, like painting any kind of arts you go through. I think it develops creativity. So which is very important. So I want to motivate anybody who is art who is in arts to join programming. I think you're going to do very well.
30 00:05:04.410 --> 00:05:08.160 Borivoj: So I finish sports gymnasium. I'm
31 00:05:08.350 --> 00:05:13.010 Borivoj: finish a master's in business economics. So I'm mostly heavily leaned
32 00:05:13.240 --> 00:05:22.919 Borivoj: most of my career in business only last 4 or 5 years. I'm I'm heavily coding and heavily programming before that I was mostly in in business realm.
33 00:05:23.700 --> 00:05:32.910 Borivoj: I started my career as a soldier in a special forces. Then I went on a cruise ships. This was my my younger days.
34 00:05:33.460 --> 00:05:41.249 Borivoj: Then I start working as a man in management, most in companies in like, let's say, top 5 to 10 people in in some big companies.
35 00:05:41.630 --> 00:05:56.030 Borivoj: And then I started working as a logistics and production manager. But I 1st time implemented back end in rust for the warehouse management system. That was. I was switching from Java to rust in that period. So it rust was a big
36 00:05:56.410 --> 00:06:00.709 Borivoj: a revelation for me in that period of time. So
37 00:06:01.030 --> 00:06:06.190 Borivoj: and then. Now, I work as a data engineer, lead data engineer, I mean in expert group
38 00:06:06.560 --> 00:06:08.949 Borivoj: here in Serbia. And, as I said, we
39 00:06:09.380 --> 00:06:11.920 Borivoj: bringing with a lot of lot of foreign clients.
40 00:06:12.250 --> 00:06:15.460 Borivoj: So yeah, a lot of work for for data
41 00:06:15.570 --> 00:06:20.229 Borivoj: people these days. Okay, so some programming languages that that I learned.
42 00:06:20.620 --> 00:06:31.889 Borivoj: And I'm also doing some courses on Udemy which you can also find data fusion like introduction for the beginners. Because I like data fusion a lot.
43 00:06:32.570 --> 00:06:38.420 Borivoj: And in December last year, I open, sourced this library illusion.
44 00:06:39.820 --> 00:06:42.979 Borivoj: Okay, so why would you use illusion?
45 00:06:43.450 --> 00:06:44.730 Borivoj: Okay. So
46 00:06:44.940 --> 00:07:02.750 Borivoj: one of the things that I was obsessing about when I was starting. This project is okay. So what I don't like about other libraries. It's this enforcement that you, forcing me to to use your library and changing your functions the way you create your process in the background.
47 00:07:02.850 --> 00:07:06.230 Borivoj: So this means that I need to remember for different libraries.
48 00:07:06.400 --> 00:07:22.749 Borivoj: I need to remember different syntax. Right? So one of the things that is very important for me was okay. How I can allow users to use their own logic to create their own queries and to get the same result. This was very, very important for me. And you're gonna see how I did this. So
49 00:07:23.220 --> 00:07:38.469 Borivoj: it has very simple data frame Api construct. If you know already. SQL, and any data frame library, it's gonna be very simple for you to understand. Okay, you're not gonna maybe know how to write immediately. But when you see it, you're gonna know what what actually, this thing does. Right?
50 00:07:39.660 --> 00:07:41.790 Borivoj: So if you want to have power of rust.
51 00:07:41.990 --> 00:07:46.160 Borivoj: of course, because it's 100% built with rust, but without the need to know. Rust
52 00:07:46.920 --> 00:07:50.120 Borivoj: thing, that the the let's say 3rd thing, that I was
53 00:07:50.310 --> 00:08:09.330 Borivoj: heavily, heavily thinking about, how can I create a library that any other problem language users can use? Of course, I'm mostly thinking about python users. Right? Because, let's say, Pandas, the most popular data frame library. It's such as, let's say, not simple, but it's
54 00:08:09.470 --> 00:08:12.150 Borivoj: it has easy constructs and easy.
55 00:08:12.250 --> 00:08:25.289 Borivoj: Let's say easy syntax to to remember without knowing python. You can actually work with pandas without knowing how to build some complex thing in python. Right? So
56 00:08:25.740 --> 00:08:30.610 Borivoj: this was like very important for me when I create this to, to, to
57 00:08:30.870 --> 00:08:33.350 Borivoj: to enable people that don't over us.
58 00:08:33.490 --> 00:08:36.629 Borivoj: and to still be able to use this use this library.
59 00:08:37.740 --> 00:08:46.959 Borivoj: Okay, we also have native pipeline scheduler. So I created the schedule with chrome scheduling. So you can automate your your jobs.
60 00:08:48.220 --> 00:08:57.439 Borivoj: And we have native reporting and dashboarding. And today I just this morning I published 2.7 point 0 version, which I added a grid to it
61 00:08:57.830 --> 00:09:14.699 Borivoj: before I had this interactive dashboards. Now, I added a grid tables which you can download to excel and Csv files as well. So this is also very important for business users to be able to export, to excel right. This is a whole whole circle, right
62 00:09:15.076 --> 00:09:29.299 Borivoj: export to give me ex export to excel button. That's very important for users. And one of the things because I saw there's a back end development in back in development. For me, it was very important to to have some sort of back end development capabilities.
63 00:09:29.790 --> 00:09:37.990 Borivoj: I consider working with the rest Apis and gathering data from Rest Api as part of the betting development. Okay, today. Data engineers work
64 00:09:38.090 --> 00:09:44.230 Borivoj: a lot of things. But my belief is that engineer do that, Etls. But
65 00:09:44.910 --> 00:09:56.389 Borivoj: without rest, Api handling mostly backend developers should do this. I believe so, but I will also want to add the rest. Api handling in the sense that we can take
66 00:09:56.680 --> 00:10:06.040 Borivoj: data from Api. Store it in Json, then use it to you. Can you can forward it to the data frame and work with it later on.
67 00:10:06.540 --> 00:10:09.048 Borivoj: Okay, so today, we have
68 00:10:09.850 --> 00:10:12.570 Borivoj: over 10,000 downloads in just one month.
69 00:10:12.940 --> 00:10:20.710 Borivoj: So one month and 20 days, let's say, yeah. So I'm very happy with this. And I'm happy that people are using this, I get some nice feedbacks.
70 00:10:21.130 --> 00:10:31.800 Borivoj: I had also nice, some nice suggestions from the community. So thank you for this. I had like much more experience. People working with us to give me some very nice, very good.
71 00:10:32.900 --> 00:10:36.490 Borivoj: very good explanations of things. How things should should break. Right?
72 00:10:36.630 --> 00:10:38.399 Borivoj: We learn every day, of course.
73 00:10:38.760 --> 00:10:48.390 Borivoj: Okay, so motivation? Why, I start. Why start this project? And what still motivates me to to work on this? Okay? So I love data fusion.
74 00:10:48.870 --> 00:10:50.689 Borivoj: When I was working with data fuse.
75 00:10:51.040 --> 00:11:08.950 Borivoj: But the thing about data fusion, you have data frame Api, and you have this SQL Api, right? So working with SQL. Api. It's good, of course, native, it's good, but it doesn't give you any sort of error while you're writing. You can write the whole sequel like this.
76 00:11:09.330 --> 00:11:26.320 Borivoj: and and only when you execute, you get some errors, and you can find where is the error? Things like this here, when I separate the concerns for for the different functions you can pretty quickly figure out like, where is the error or find it? It's much easier to work with right, because it has this
77 00:11:26.560 --> 00:11:33.070 Borivoj: sort of has this SQL execution kind of a type, but still has this data frame Api function
78 00:11:33.190 --> 00:11:38.569 Borivoj: kind of a look. So it's it has like I think best from both
79 00:11:38.860 --> 00:11:46.820 Borivoj: Apis, let's say. But one of the things that is very important is that mixing different Apis from data fusion
80 00:11:47.470 --> 00:11:52.740 Borivoj: forced me to rewrite the whole library twice.
81 00:11:53.370 --> 00:12:06.080 Borivoj: So I will. I will explain why in just a bit. Okay. So another thing. So this new norm in today's data world is, let's build something with rust. And let's create python bindings. And
82 00:12:06.200 --> 00:12:10.459 Borivoj: let python users use it right? There's like a new norm today in data world.
83 00:12:10.660 --> 00:12:15.339 Borivoj: Also, data fusion is doing this, creating this wrapper for python.
84 00:12:15.890 --> 00:12:24.120 Borivoj: just for the fact that should be easier to use. Right? That's 1 of the goals of them, which I also have for this library
85 00:12:24.570 --> 00:12:28.100 Borivoj: same thing with polars. They start with the rust Api, which was
86 00:12:28.680 --> 00:12:31.409 Borivoj: pretty hard to use for anybody who is at
87 00:12:31.750 --> 00:12:46.579 Borivoj: from Russ world. Then they create very nice Api, right? So now everybody can understand this like not everybody but who who works with with the data frame. Apis can easily understand what is going on here. Right? They ramp all their functions in
88 00:12:46.980 --> 00:12:48.810 Borivoj: in Python, let's say, and
89 00:12:48.980 --> 00:12:56.059 Borivoj: you can now see and understand much, much easier. So this like a new norm that is happening in data role, I would say.
90 00:12:57.720 --> 00:12:59.240 Borivoj: okay, so
91 00:13:00.350 --> 00:13:15.059 Borivoj: as a data engineer working in Microsoft fabric. I work a lot in notebooks, of course, and what usually can happen in one notebook. We can have price part cell. Then we have SQL cell. Then you have price, part cell SQL. Cell.
92 00:13:15.700 --> 00:13:18.189 Borivoj: They they go on and on. So
93 00:13:18.530 --> 00:13:23.580 Borivoj: I would like to have something. Why is this? Because usually SQL. Have
94 00:13:23.810 --> 00:13:36.609 Borivoj: their advantages past park here, its own advantages in flexibility and stuff like this. So when usually, when I come to some project like in middle of the project, or you know, some project is already developed.
95 00:13:36.710 --> 00:13:46.750 Borivoj: You you come to the notebook which has price per sequel price per sequel. Right? So my goal was to create something that would have SQL.
96 00:13:46.870 --> 00:13:52.189 Borivoj: Query engine in the background to be executed like an SQL. But still to have
97 00:13:52.460 --> 00:14:08.629 Borivoj: some sort of flexibility like like Price Park. So this this was when the whole idea came from to to like to somehow mix these 2, to have one library that would replace both of these and to be able to work with it.
98 00:14:10.630 --> 00:14:13.610 Borivoj: Okay, so goals, I have a lot of goals
99 00:14:13.710 --> 00:14:21.820 Borivoj: when I was starting this. So what was, as I already said, to create simple data frame Api, that can be understand and used by anybody
100 00:14:22.400 --> 00:14:40.080 Borivoj: to make rust fields approachable to anybody who is not from rust right? So already said you should not be worried about. If you don't know rust, it's not. It's not a big problem, because you will see now how much of rust you need to to know, to be able to use this very, very little.
101 00:14:41.130 --> 00:14:51.819 Borivoj: So, as I said, to allow users to create their own query constructions. So if you want to start with, join, start with, join. If you want to start with, select start with select. You will always get the same.
102 00:14:52.000 --> 00:14:54.090 Borivoj: you will always get the same result.
103 00:14:54.190 --> 00:15:01.379 Borivoj: So also create betting capabilities. That was a part of Rest Api that I was talking about
104 00:15:01.910 --> 00:15:04.819 Borivoj: and to create reporting capabilities.
105 00:15:05.250 --> 00:15:09.960 Borivoj: Okay? So, as I said, we cannot talk about delusion without data fusion.
106 00:15:10.230 --> 00:15:12.169 Borivoj: The king of all kings. Right?
107 00:15:12.360 --> 00:15:21.579 Borivoj: This, what I believe, I think data fusion has has a great potential. But, as already showed, maybe in this previous slides.
108 00:15:22.080 --> 00:15:31.179 Borivoj: maybe it's not so easy usable by the data frame users like, or you need to know row SQL, or this data frame. Api.
109 00:15:31.370 --> 00:15:37.450 Borivoj: It's little bit, you know, not so easy to use. So what I did
110 00:15:38.150 --> 00:15:43.889 Borivoj: for this library after I rewrote it a couple of times. And why did I rewrote it? Because
111 00:15:44.550 --> 00:15:51.670 Borivoj: data frame Api expression Api SQL Api. I try to mix all together like to use the best
112 00:15:51.820 --> 00:16:04.549 Borivoj: from all Apis. And then the query engine was so confused that I couldn't. I couldn't execute half of the functions, and in order to execute most of the functions I would need to rewrite them from the scratch.
113 00:16:04.780 --> 00:16:07.030 Borivoj: and that would take me probably
114 00:16:07.390 --> 00:16:11.979 Borivoj: whole year. I don't know how much. But it would be a lot a lot of time. So
115 00:16:12.700 --> 00:16:17.900 Borivoj: 3rd time, when I was start rewriting the full library. Okay, I said, I'm gonna lean on on sequel
116 00:16:18.070 --> 00:16:30.519 Borivoj: reference. I'm gonna lean on just some pure SQL, and I'm gonna build a top of that. So all of the functions are going to execute any of these now functions you can see on screen.
117 00:16:33.530 --> 00:16:47.598 Borivoj: Okay, let's meet illusion. So how illusion came from. So I start to build, maybe year and a half 2 years ago, some project that should be similar, like, excel like, but just like
118 00:16:48.530 --> 00:16:51.910 Borivoj: to use SQL. Instead of these excel functions.
119 00:16:52.190 --> 00:17:20.179 Borivoj: This I couldn't. I didn't have time at that time. I didn't have enough time on my hands to do this. So this this project fails. And then, when I started learning about data fusion, and I already have this project for excel. Somehow this illusion came to my mind like this is the way I create this name. Illusion, excel plus illusion. Illusion, right? Not very original. But this just came up to my mind.
120 00:17:20.690 --> 00:17:29.099 Borivoj: So yeah, okay, so that's it. So why? So
121 00:17:29.220 --> 00:17:38.649 Borivoj: what is the illusion? It's just like illusion, right? And because it's built on something like data fusion. What you're actually doing, you are triggering
122 00:17:39.440 --> 00:17:44.740 Borivoj: SQL per engine in the background. So it's just basically like an illusion like a wrapper around the
123 00:17:45.050 --> 00:18:06.039 Borivoj: around this data furion. So this, I think, comes together like you're using abstraction over something that is very powerful, very strong. So it's just like it's like, illusion. Illusion. So 1st thing that I thought is a chameleon, because it can change colors and still do the same thing. Right?
124 00:18:07.880 --> 00:18:10.650 Borivoj: Okay, so prerequisite to use this library.
125 00:18:10.830 --> 00:18:31.059 Borivoj: So you need to have at least 1.8 81 ras. Because of the Delta leak, I implemented Data Lake, which is not part of data fusion. So Delta Lake requires 1.8 1 ras, because of the writers to be able to write Delta Leak, you need to have at least 1.8 1
126 00:18:31.520 --> 00:18:32.770 Borivoj: version of rust
127 00:18:34.454 --> 00:18:41.459 Borivoj: latest, and the great solution is 2.7 point O point on from this morning and Tokyo 1.42 for.
128 00:18:42.340 --> 00:18:46.570 Borivoj: And you need to enable features rt, primetime multi-trend.
129 00:18:47.190 --> 00:18:52.099 Borivoj: Okay? So what I was talking about not need to know. Rust.
130 00:18:52.260 --> 00:18:56.129 Borivoj: All that you need to know from the rest is literally
131 00:18:56.340 --> 00:18:59.299 Borivoj: main function. How to write this main function.
132 00:18:59.680 --> 00:19:07.059 Borivoj: Okay? So this user illusion payload, we're gonna give you everything that you need from this library, literally everything.
133 00:19:07.570 --> 00:19:12.389 Borivoj: And then when you create the main function should be asynchronous, obviously, because this is
134 00:19:12.853 --> 00:19:23.850 Borivoj: made us with the stickers Runtime Tokyo, which you put on top of this main function, as you can see Tokyo Main. I'm talking like this because I'm assuming there may be some rust
135 00:19:24.230 --> 00:19:29.310 Borivoj: or the some python or some other other listeners. So
136 00:19:29.860 --> 00:19:39.180 Borivoj: just you literally just need to know this 1, 2, 3, 4 lines of code from the rest. Everything else you're gonna see. It's pretty much similar to any other.
137 00:19:39.390 --> 00:19:41.029 Borivoj: any other data frame.
138 00:19:42.620 --> 00:19:44.989 Borivoj: Okay, readers and writers.
139 00:19:45.650 --> 00:19:56.699 Borivoj: So to create a data frame, you don't need to establish schema from version 0 point 2.5 I created dynamic schema
140 00:19:57.310 --> 00:20:01.989 Borivoj: in in to to be able to infer schema dynamically just from the
141 00:20:02.110 --> 00:20:18.320 Borivoj: file extension. So if your file extension is Csv. Per K. Json, or it's just a folder name. It's gonna know which file it's coming in. And it's gonna pass it like this. And it's gonna establish schema by peeking into the file. So
142 00:20:18.460 --> 00:20:34.670 Borivoj: there is like a function that is peeking into your 100 lives of lines of of file, takes the data, creates a schema for you already knows which data type, that data file is coming in and using the proper loader for this.
143 00:20:35.520 --> 00:20:54.159 Borivoj: So what you need to supply for the creating. The new data frame is just the path to your file, as you can see, and the alias, alias, is very important. If you're importing multiple files and you're going to need to join them later on. You need to provide, alias because you're going to see, you need to use, alias Dot.
144 00:20:54.290 --> 00:20:58.449 Borivoj: for your name, very similar, like like you when you're using SQL
145 00:21:00.890 --> 00:21:04.550 Borivoj: to just the path and the and the areas is very important.
146 00:21:05.040 --> 00:21:29.909 Borivoj: And this is a synchronous task of force. So you need to add a weight in rust. It's it's like in any other program language you you when you have asynchronous Runtime, you need to add a weight keeper. Right? This question mark in rust is just propagating the error. So if you made, if there is some mistake that you made, or something that you don't do right. This is gonna return you an error. It's gonna
147 00:21:30.240 --> 00:21:36.200 Borivoj: evaluate against the function that is running in the background. And I think that I provide very, very nice
148 00:21:37.021 --> 00:21:48.669 Borivoj: errors. I did create all the custom errors. So you're gonna get some nice messages like, okay, check this. Okay, check that. So I think I cover most of the of the thing that you can make a mistake with.
149 00:21:49.000 --> 00:21:51.860 Borivoj: So I think that are very helpful. The error messages.
150 00:21:53.060 --> 00:22:00.309 Borivoj: Okay, so readers, so I also have database readers. So you're gonna see now. So for the database
151 00:22:00.840 --> 00:22:06.960 Borivoj: for the other things you're gonna see? Now, you need to have to to provide the connection which is
152 00:22:07.510 --> 00:22:20.240 Borivoj: create from the string. And you need to provide the SQL. Query. So besides the connection for the database, we also have the connection from the blob storage. Currently, I'm still trying to implement for S. 3
153 00:22:20.450 --> 00:22:35.269 Borivoj: for aws still working on this. But currently you can. You can take the debt. Besides the files you can take from databases and from azure blob storage databases that I covered currently are my SQL
154 00:22:35.950 --> 00:22:39.709 Borivoj: and I covered the podcast skills. So these 2.
155 00:22:39.920 --> 00:22:52.120 Gabor Szabo: Sorry. There's a question. I think I don't know. I missed it earlier, just out of curiosity, curiosity! I can't see this English in word in English which create did you use for defining your errors?
156 00:22:52.520 --> 00:22:53.710 Gabor Szabo: This error.
157 00:22:54.620 --> 00:22:58.570 Borivoj: No, no. So data fusion and delta leak
158 00:22:58.670 --> 00:23:07.699 Borivoj: these 2. These 2 crates have their own error, data, fusion, error, and delta error. So on top of these errors I created a custom error.
159 00:23:08.040 --> 00:23:19.430 Borivoj: You can. You can go. It's very easy. The project is very simple to navigate through on Github. So you can check all the whole implementation literally, like in one file for errors.
160 00:23:20.080 --> 00:23:42.509 Borivoj: So I just wrap, wrap those errors and create my own errors which gonna come from? Is it the joining? Is it the select? Is it? Is it a string function? It's gonna target different parts of your code. And it's gonna evaluate against these parts. Gonna tell you. Oh, okay, check this. Maybe it's this problem. Oh, check this. Maybe joining is the problem. So things like this.
161 00:23:42.960 --> 00:23:46.670 Borivoj: Of course it's not a bulletproof error system, but
162 00:23:47.113 --> 00:24:03.470 Borivoj: I still don't work on it. But I'm trying to target each part. This is this is why, this is very good when you separate functions for different parts of execution, that you can target errors much better to send to users what? What is actually the problem.
163 00:24:04.630 --> 00:24:12.669 Borivoj: Okay? So I said, about my secret postgres azure. Okay? So for the azure you need price, sas token.
164 00:24:13.030 --> 00:24:38.249 Borivoj: And blob URL, your container. URL. This is going to implementation. So with the Sas token and with the with the blob URL and filtering is optional. You see some. You can. You can put none if you don't want. If you just want to grab everything from the container, if you want to grab certain part of the path. You just right there. Or if you want to just take a file, you write the file name is going to filter out
165 00:24:38.490 --> 00:24:50.450 Borivoj: everything from your container. Gonna find this file and download this for download, this for you. And of course, you need to provide areas for this table, because this is gonna directly like, evaluate this for you
166 00:24:50.920 --> 00:24:52.480 Borivoj: and add, alias.
167 00:24:53.540 --> 00:24:55.980 Borivoj: So yeah, URL and Sas token.
168 00:24:56.130 --> 00:25:25.920 Borivoj: Okay? So Json readers, this was one of the most difficult implementations I did here because Json readers from data fusion. And this any Json reading is mostly leaning through K and value pairs, very little implementations, and very, I don't know I couldn't find any crate that is, can read erase objects and flatten files.
169 00:25:26.030 --> 00:25:28.780 Borivoj: Usually you need to create your own structs
170 00:25:28.970 --> 00:25:32.050 Borivoj: and then, like deserialize the data.
171 00:25:32.350 --> 00:25:43.480 Borivoj: So this was very, very challenging, not just how to implement, but to make somewhat performant, because flattening fast, you know, it can be very challenging
172 00:25:43.760 --> 00:26:08.080 Borivoj: for performance. So I implemented Byte stream reading for this. And I think it's basically performed. But just for you to know, you can flatten if you have field and arrays field and objects. Illusion. Gonna flatten this file for you and gonna fill like gonna fill all these places that usually you will need to create struct for, or some more complex type
173 00:26:08.270 --> 00:26:10.789 Borivoj: to to be able to deserialize. But
174 00:26:11.310 --> 00:26:19.210 Borivoj: I spend a lot of time on this, so you can easily easily flatten files. And of course, key value pairs are the most performant. It just
175 00:26:19.390 --> 00:26:21.000 Borivoj: like blazing fast
176 00:26:23.380 --> 00:26:32.140 Borivoj: writers. Okay, so we have market writers. So we can write to a parquet. We can write to Csv files, and we can write to Delta
177 00:26:32.560 --> 00:26:46.119 Borivoj: this for the for the local file. So per K, you have overwrite and append mode. Csv. You have also write an append mode. But for the Cs you need to provide CC right options. This, this is mandatory without.
178 00:26:46.120 --> 00:26:49.890 Gabor Szabo: Second, just a second. I think Ray wanted to ask something. I don't know if.
179 00:26:49.890 --> 00:26:52.156 Ray Lutz: Yeah, if you don't mind
180 00:26:52.610 --> 00:26:55.070 Gabor Szabo: No, no! Feel free just to jump in and start asking.
181 00:26:55.300 --> 00:27:06.370 Ray Lutz: Well, I sometimes it's better to hear hear more, just to make sure that he wasn't going to cover it. But in terms of Json. Some databases.
182 00:27:07.245 --> 00:27:14.700 Ray Lutz: Do offer additional Json interpretation. I know postgres is is
183 00:27:14.850 --> 00:27:19.160 Ray Lutz: really good at it. Even SQL. Lite
184 00:27:19.370 --> 00:27:23.829 Ray Lutz: has some additional Json functions where you can iterate through
185 00:27:24.100 --> 00:27:29.420 Ray Lutz: Json structures that may be contained in one field, for example.
186 00:27:29.740 --> 00:27:38.390 Ray Lutz: but because you're basing everything on on the fusion
187 00:27:39.160 --> 00:27:42.100 Ray Lutz: data fusion. Or what is this?
188 00:27:43.290 --> 00:27:49.379 Ray Lutz: the Apache data fusion? It doesn't offer that. And so you're not offering anything other than what they offer. Correct.
189 00:27:49.380 --> 00:28:00.180 Borivoj: So for the databases. I'm not leaning on data fusion execution. I'm targeting through all. Dbc, you're gonna see later. So you need to create all the BC connections. So you can, you can create.
190 00:28:00.957 --> 00:28:16.839 Borivoj: You can. You can take 2 paths. You can just target directly database. Write your SQL. And put all the pressure on the SQL. Engine like Postgres or Mysql is gonna wrangle data and is gonna return the flat
191 00:28:16.890 --> 00:28:36.509 Borivoj: form for you. So in in form of columns and rows. So you put all the pressure directly on the on the query engine of there. So this is not coming through data fusion. So what comes from data fusion is once you retrieve the data from the database, then when you push the data into the data frame, then data fusion
192 00:28:36.640 --> 00:28:43.030 Borivoj: bronze, the query that you wrote right? So the whole point of the connectors is to take the data
193 00:28:44.120 --> 00:28:50.859 Borivoj: from any source and then to push it into the data frame. And then you work with the data fuse. So.
194 00:28:50.860 --> 00:29:09.709 Ray Lutz: Okay, okay. So so that's important to understand, are you gonna get to this? Because I don't want to jump the gun if it's coming later. So basically, basically you can work with other databases. And the way you do that is you're 1st bringing the data in from the database into your in memory data fusion, implementation.
195 00:29:09.710 --> 00:29:10.190 Borivoj: Yes.
196 00:29:10.190 --> 00:29:13.409 Ray Lutz: And then when you create a data frame
197 00:29:14.378 --> 00:29:21.360 Ray Lutz: it's creating it within the data fusion namespace, for example.
198 00:29:21.520 --> 00:29:22.530 Ray Lutz: And
199 00:29:22.690 --> 00:29:30.090 Ray Lutz: when you create a data frame. Normally in these SQL. Things. They haven't that each table has a name.
200 00:29:30.410 --> 00:29:32.380 Ray Lutz: So does does your.
201 00:29:33.670 --> 00:29:43.180 Ray Lutz: Does the name? Do you name the tables, or does the name wind up being just like Df. For example, is it the local name.
202 00:29:43.180 --> 00:29:50.729 Borivoj: Yes, yes. So so yes, that's a good. That's a good question like this part of implementation. So I create a very
203 00:29:51.100 --> 00:29:55.480 Borivoj: crazy aliases. That's, I assume nobody will hit
204 00:29:55.790 --> 00:30:04.919 Borivoj: when you create a query, but you need to apply your own, alias. So this might Crazy, alias will be replaced with your alias.
205 00:30:05.150 --> 00:30:24.450 Borivoj: So that's how I I overcome this. But yeah, you don't need to worry. I'm pretty sure you're not gonna hit the alias that I wrote for the when you trigger the the database. But later on, when you provide your own alias is gonna be replaced. You're not gonna feel this. You're gonna know this, you just supply your own alias, and then you work with it. Later on.
206 00:30:25.130 --> 00:30:27.469 Ray Lutz: Okay, I'll let you continue. Thanks.
207 00:30:27.470 --> 00:30:35.569 Borivoj: No problem. So for the Cs writing. So you need to provide the limited escape for double quote and null value. How you want to replace the new values.
208 00:30:36.040 --> 00:30:52.329 Borivoj: And then it's gonna work. So he has headers. It's true. So you need to work with headers. I don't think anybody in data will work with Csv without headers these days, maybe. But I disabled this. So if you want to work with Csv, you need to have headers right?
209 00:30:53.000 --> 00:31:04.639 Borivoj: And I just wanna say for the for the partition. I don't know if you can see this part that, says Delta, but it's got it's got to be this thing on on my screen. I don't know if you can see it.
210 00:31:05.330 --> 00:31:11.419 Borivoj: because I can see the the screens of of who is on the on camera. But anyway, so for for delta
211 00:31:11.690 --> 00:31:19.460 Borivoj: partitioning. So you also have a very important thing to know when you want to partition the table. You see, it's optional.
212 00:31:19.570 --> 00:31:27.190 Borivoj: It's a 3rd argument for partitioning. So I I wrote here some vector and the name of the of the column and dot into
213 00:31:28.990 --> 00:31:37.449 Borivoj: it's very important for you to know you're not gonna be able to retrieve this column anymore. Once you partition the data on. So my recommendation is create new column
214 00:31:37.980 --> 00:31:48.799 Borivoj: which you wanna partition the delta table on. So if you wanna partition on date or on integer, mostly, it's dates or some integer create new column.
215 00:31:49.300 --> 00:32:04.970 Borivoj: however, you want to call it, call it, and then partition your data. And that's very important, because once you want to read back the data, this column is not going to appear, because it's it's sucking in partition for appending. When you want to append on that table, make sure you have the same
216 00:32:04.970 --> 00:32:21.480 Borivoj: column name and the same column. We're gonna that you already partitioned on to have this same column on this new new data that is coming in because you're gonna otherwise, it's gonna fail. You need to have the same column name and the same data type is date, or is integer right
217 00:32:21.480 --> 00:32:28.749 Borivoj: to be able to partition and to be able to append on the Delta table. So that's very important for for for you to know, this is current
218 00:32:30.264 --> 00:32:52.110 Borivoj: currently how it's implemented. Okay? So just wanna say, if anybody want to implement create data frame, just for you to know, overwriting is easy. Appending is very hard to implement very hard. If anybody wanna try this to do, it's very hard, because you need you cannot lean on any implementation is coming. Because at this point, you're gonna
219 00:32:52.620 --> 00:32:58.250 Borivoj: have so many different intricacies to work on. And you, you're gonna need to implement.
220 00:32:58.950 --> 00:33:24.469 Borivoj: you need to work against your record batches. And you're gonna need to prick a lot a lot on performance. This is much much harder to append than to, and then to overwrite for overwriting. I I usually use the native overwriters of the data fusion for the Csv and packet. But appending it's it's not working. So you need to. So just if anybody will try to do something like this. It's much harder to append than to overwrite.
221 00:33:24.920 --> 00:33:42.079 Borivoj: Okay? So to azure blob storage writing is currently only supported in parquet. Why is this? Because at my current one of my current projects I need. I had this requirement to grab the data from Api and to write in the into azure to park it like more
222 00:33:42.650 --> 00:33:48.810 Borivoj: did all like file types of work. Come, I will implement for Json writing as well, and Csv.
223 00:33:49.890 --> 00:34:02.289 Borivoj: for now only only the per key file you can write. So this is a small example. So let's say you have some data custom data frame. Df, dot select here is just like an example. Of course it's can be much more complex.
224 00:34:02.770 --> 00:34:08.630 Borivoj: Then you you state your URL to the your container folder, sas token.
225 00:34:08.830 --> 00:34:27.029 Borivoj: And you use this data frame data. You see above what is called query dot illusion. It's very important. Illusion function is like an execute function in other data frame. So illusion is actually evaluating your query. And it's creating a new
226 00:34:27.909 --> 00:34:47.689 Borivoj: alias for the later on usage. If you want to use multiple like in same execution. You want to use the same. Let's data source. And you want to create multiple queries until you come to your final query. So it's very important to use illusion, evaluation. Before that you don't need to use it. You can just change, change and then
227 00:34:47.830 --> 00:34:52.590 Borivoj: make sure to use illusion. It's like an execution on top of your all of your queries.
228 00:34:52.889 --> 00:35:14.540 Borivoj: and then you can see this use right parquet to azure with Sas. I try to create all the function as as more as most logical, as I think. Like to remember. I'm writing per key to azure with us. So that's how I I name my function somehow. For me it's easier to remember. I don't know if this is very practical. But yeah.
229 00:35:14.690 --> 00:35:31.070 Borivoj: maybe somebody gonna disagree with me. But for me to azure with Sas. And this how I call this. So you have 2 2 modes, overwrite and append, and, as you can see, you provide your URL to to the storage and the sas token.
230 00:35:31.450 --> 00:35:56.619 Borivoj: So for writing, it's very important to make sure for such token that you have writing permissions to check this. Some of my colleagues are persistently trying to write with Sp. Tokens without writing and creating permissions, so make sure to to have this to have this insight. So writing is set to default. Compression is snappy 2.0 per kit, and the buffer size are put 1Â GB.
231 00:35:56.790 --> 00:36:14.650 Borivoj: I can extend this, but I think 1Â GB is. It's a it's a big parquet file. So like I, I said, this limit this mostly for also for my purposes. If something goes wrong with some Api not to not to, you know, make a big bill
232 00:36:14.840 --> 00:36:16.129 Borivoj: for my company.
233 00:36:16.980 --> 00:36:22.057 Borivoj: So data wrangling. So we're gonna now cover a little bit of syntax and
234 00:36:22.860 --> 00:36:35.219 Borivoj: a bit of design. So we're gonna go through different functions that I create. And why I decided to to use some things in some functions and take it out from from other ones.
235 00:36:35.750 --> 00:36:46.849 Borivoj: So when we talk about select select has a case insensitive. S, so can you use capital S or S. Small? You can see here.
236 00:36:47.620 --> 00:36:58.099 Borivoj: You can, of course, select star. You can select everything, or you can select, count everything and and etcetera. So from this 1st glance.
237 00:36:58.220 --> 00:37:03.520 Borivoj: you already, I think you understand what is going on here. I think you already realize that I created
238 00:37:04.180 --> 00:37:23.610 Borivoj: functions which you're gonna give you greater flexibility. But you're gonna still be able to use this sort of a SQL syntax, because, of course, we are targeting data fusion Sequel SQL engine in the background. So this is kind of a design like, maybe you're not gonna like it. Maybe you're gonna like it. But
239 00:37:23.980 --> 00:37:34.330 Borivoj: we're gonna see now. So for numerical operators. So currently operators operators that are supported is plus minus multiplication division, the mode.
240 00:37:34.520 --> 00:37:45.545 Borivoj: So later on, you're gonna see in which functions you should use. Which currently, you can see a small example here of how you can use the filter filter is basically like,
241 00:37:46.380 --> 00:37:47.380 Borivoj: where
242 00:37:47.570 --> 00:37:56.090 Borivoj: where in in SQL, so you're filtering a pre aggregated data with filter having which you're gonna see later on, is filtering
243 00:37:56.750 --> 00:38:07.599 Borivoj: data after aggregation like very similar like you have in the in, in, the, in this, in the sequel. So for order by here, it's a single single
244 00:38:08.200 --> 00:38:19.410 Borivoj: argument, because you also have order by many which you can supply multiple multiple columns to order by. Currently, I have for ascending and descending. I have true and false
245 00:38:20.390 --> 00:38:24.459 Borivoj: still thinking how to to implement this, maybe just to to be able to to add
246 00:38:24.680 --> 00:38:32.099 Borivoj: a SC, or DESC. Instead of true false, but currently is true. False, true is ascending false is descending
247 00:38:32.260 --> 00:38:55.159 Borivoj: and limit. We all know what is limit is just you're limiting the the amount of of roles you, wanna you wanna see, and then you evaluate with illusion. As already said, you added a new alias, because this, if you want to use it further, you can. You can evaluate with illusion function, and then, later on, you can use it in other. In other query
248 00:38:55.340 --> 00:39:01.800 Borivoj: and display function. We already know what is display function which is gonna display the the data frame on the on the console for you.
249 00:39:02.430 --> 00:39:05.955 Ray Lutz: So on this one before you go on the
250 00:39:08.050 --> 00:39:15.219 Ray Lutz: The 1st part, when you're creating this variable called num OP. Sales.
251 00:39:15.750 --> 00:39:20.939 Ray Lutz: those the functionality there just pretty much constructs the query right?
252 00:39:21.340 --> 00:39:26.086 Ray Lutz: And then in the the let down below.
253 00:39:27.120 --> 00:39:33.510 Ray Lutz: that actually essentially executes the query. Is that the way to think of it?
254 00:39:33.700 --> 00:39:41.970 Borivoj: Is, is exactly the way you should think about it. So you change. Change, then, with illusion. You are
255 00:39:43.210 --> 00:39:53.710 Borivoj: what is the point in illusion. We have another function in the background that is called construct sequel, which you're never gonna see in implement for usage. But it's in the background.
256 00:39:54.460 --> 00:40:08.800 Borivoj: So the the way this functions, and why you can actually write in any sequence. The the function says, like select filter, you can like literally mix up and down any of these functions is going to work the same, because
257 00:40:08.980 --> 00:40:17.420 Borivoj: when you evaluate allusion, it's constructing in the background the SQL. Query in the order which query needs to be constructed.
258 00:40:17.420 --> 00:40:18.200 Ray Lutz: I see.
259 00:40:18.200 --> 00:40:24.850 Borivoj: So now you need illusion to evaluate your credit. You wrote against
260 00:40:25.290 --> 00:40:35.860 Borivoj: SQL. Construct in the background. And that's how I implemented this. Okay, use whatever you want to use, and however you want to use it is going to be, give you the same result in the background. So this is
261 00:40:36.030 --> 00:40:36.880 Borivoj: like.
262 00:40:37.750 --> 00:40:41.729 Ray Lutz: So, except except for the that is there.
263 00:40:42.090 --> 00:40:57.380 Ray Lutz: In other words, I'm trying to figure out what is the benefit of using illusion. Certainly you get that benefit because you don't have to get these in the right order, and you're checking that in creating the query properly.
264 00:40:57.600 --> 00:41:01.479 Ray Lutz: But pretty much you have to know about queries and how they all work
265 00:41:02.202 --> 00:41:07.689 Ray Lutz: very much like a regular SQL. Query. Right? I mean, yes, very much.
266 00:41:07.690 --> 00:41:11.290 Ray Lutz: Yes, very much similar. But you're gonna see some advantages.
267 00:41:11.850 --> 00:41:29.550 Borivoj: As I said, it's it's it's I'm trying to make a i i already made it a mix of pie spark and sequel, so to have flexibility to write the way you want to write. You don't need to use like, select everything from where this group by this order, this right? You don't need to write it like SQL,
268 00:41:29.780 --> 00:41:36.889 Borivoj: but it's basically like a sequel, right? But it just you. Can. You have this flexibility to write any way you want to write it. So it's like
269 00:41:37.400 --> 00:41:37.940 Borivoj: the one.
270 00:41:37.940 --> 00:41:40.289 Ray Lutz: Okay, and then, and the result of this.
271 00:41:40.620 --> 00:41:44.190 Ray Lutz: So when you do an execute, then
272 00:41:44.360 --> 00:41:49.569 Ray Lutz: would you get out of that a, an object which is.
273 00:41:49.690 --> 00:41:52.210 Ray Lutz: you get this object? None ops res.
274 00:41:52.770 --> 00:42:00.550 Ray Lutz: and then you say, dot display now is that none ops res is that like a proxy object that is.
275 00:42:01.150 --> 00:42:03.210 Borivoj: It's new data framework. It's new data.
276 00:42:04.200 --> 00:42:11.620 Borivoj: So against this, Num Ops rest. Now, you can change more. So, for example, you aggregated, aggregated something.
277 00:42:11.620 --> 00:42:11.990 Ray Lutz: Okay.
278 00:42:11.990 --> 00:42:36.559 Borivoj: You. You join 10 tables, or whatever, and you evaluate Num. Ops results. Then this, this variable is going to hold all result of previous square. Then you can work later on. If you're on a union or something, or append, or whatever you can use against this. So because it's storing a data frame which is evaluated with illusion. Right? So.
279 00:42:36.560 --> 00:42:42.949 Ray Lutz: Is that always a new frame in your mind? Or do you sometimes return the same one like if it's a join?
280 00:42:43.220 --> 00:42:47.959 Ray Lutz: Well, maybe we're gonna get to join. So I'll wait. Okay, thanks. I'll stop there.
281 00:42:48.060 --> 00:42:48.585 Borivoj: Okay.
282 00:42:49.640 --> 00:42:53.050 Borivoj: So filtering as I said, you can filter many
283 00:42:54.780 --> 00:43:05.490 Borivoj: very important thing which I'm still dribbling in my mind. I wanna implement inline parameters. I gonna talk about this at the end. So current implementations and development
284 00:43:06.416 --> 00:43:19.979 Borivoj: but hopefully, if I figure it out the way I wanna figure it out so soon. We should have also be able to use inline parameters meaning you can create some scalar value or just
285 00:43:20.110 --> 00:43:27.338 Borivoj: any value in in. You want to store in a in a variable, and then you can. You should be able to evaluate against
286 00:43:27.800 --> 00:43:55.959 Borivoj: your data frame data frame column. So this couldn't what is coming. But currently you have filter which saw before, and you have filter many. Just make sure to make it in square brackets, because it's like like a list writing in in rust. I'm talking like this because of the I assume python users. So you put in square brackets and each inch filter you separate with a with a with a with a regular like close bracket. So this, how you you create multiple filtering
287 00:43:55.960 --> 00:44:00.160 Borivoj: again. As I said, filtering is used for pre-aggregated
288 00:44:00.160 --> 00:44:24.850 Borivoj: values. So having is used after aggregations. But you're going to see now. Order by we covered and everything of this we covered already. So scalar functions, so scalar functions are used exclusively in select function, they will not work in other functions like aggregation. And like string functions that I created. You're gonna see now. So you can use scalar functions directly in in select.
289 00:44:25.670 --> 00:44:30.070 Borivoj: Of course, you can use this nested, nested scalar functions as well
290 00:44:30.290 --> 00:44:36.440 Borivoj: filtering recovery. But we cover already so aggregating functions, so aggregating functions.
291 00:44:36.560 --> 00:44:56.580 Borivoj: of course, can can use nested functioning so round, average, absolute, but also can use operative based aggregations, as we also see previously in the in the select function. But here, if you use aggregate function, you can also, like illusion, is able to evaluate your
292 00:44:56.690 --> 00:44:59.560 Borivoj: operator operator based aggregations as well.
293 00:45:00.060 --> 00:45:21.090 Borivoj: So just something to keep in mind. And we come to the new function group by group by have 2 variants you're gonna see? Now have group by and group by many. I try to use the like similar logic for everything. Is it single? Or it's many group by or group by many right field filter, many order by order, by many. So everything is like this
294 00:45:21.470 --> 00:45:34.629 Borivoj: and usually the same. So currently group by, you obviously need to put all the non aggregated columns in order for this to work. If you if you miss some non aggregated column, it's gonna fail. You're gonna you're gonna get error message
295 00:45:34.930 --> 00:45:36.820 Borivoj: for a group. By all.
296 00:45:37.050 --> 00:45:45.920 Borivoj: it's automatically evaluating all of your non aggregate column. So I usually just use group by all if I need something. But
297 00:45:46.120 --> 00:45:50.859 Borivoj: of course, if it's small in enough example, you can use group by as well.
298 00:45:52.610 --> 00:45:55.630 Borivoj: Okay. So now we come to the more complex query.
299 00:45:56.000 --> 00:46:04.289 Borivoj: starting so select aggregation, filtering group by all, as you can see below, so group by all is gonna evaluate all of your
300 00:46:04.720 --> 00:46:24.139 Borivoj: non aggregate functions. And then it's going to give you the the right result and ordered by many, as you can see here is, similarly used like any other multiple argument function. So you put square brackets and in individual brackets, you put the the conditions right. So
301 00:46:24.320 --> 00:46:28.949 Borivoj: in this case, total billable, false, and Max, absolute billable value.
302 00:46:29.450 --> 00:46:36.170 Borivoj: and the rest is the same. You use illusion, relation, and and you you break on it as as usual.
303 00:46:36.280 --> 00:46:40.139 Borivoj: Okay, so currently supported functions are these
304 00:46:41.230 --> 00:46:50.939 Borivoj: so for aggregations and for scalar functions? So this what is currently supported, I think, is more than enough. But if some somebody have some suggestions.
305 00:46:51.750 --> 00:46:56.800 Borivoj: Please let me know, because you would think, Okay, I can use probably everything from sequel. No, you cannot.
306 00:46:57.660 --> 00:47:01.079 Borivoj: This is still restricted, because I'm passing this
307 00:47:01.240 --> 00:47:07.109 Borivoj: very heavily in the background, each of these each of these words, each of these bracket.
308 00:47:07.220 --> 00:47:27.010 Borivoj: if everything from this, it's heavily passed in the background. So like, if I like, you know, when I say, Okay, this is supported literally, this supported. So if anybody like needs more, please, so you can easily find me, and if you need some more functions. But I think this cover most of the what is needed.
309 00:47:27.590 --> 00:47:36.850 Borivoj: Okay, so joins. So we have different type of join. So we can join 2 data frames for joining 2 data frames we're gonna usually use join.
310 00:47:37.360 --> 00:47:53.419 Borivoj: So you're gonna very similarly, like you in any other data frame libraries. You provide the this, for example, df, sales, this some evaluated data frame from before and then. Now you have another data frame that you had
311 00:47:53.640 --> 00:48:01.039 Borivoj: somewhere above in your code. Now you want to join these 2. So you have Df sales and you have Df customers, and then you provide the joining column.
312 00:48:01.450 --> 00:48:07.190 Borivoj: and of course you you provided the the type of joining. Is it in there left, or or whatever?
313 00:48:07.600 --> 00:48:22.250 Borivoj: And the rest is the same aggregation group by having okay. So having this is single having, and you provide total quantity greater than 10. As you can see, it's evaluating against aggregated function.
314 00:48:22.430 --> 00:48:30.089 Borivoj: This very important. So having filtering aggregations filter filtering before aggregations. Okay.
315 00:48:31.680 --> 00:48:37.240 Borivoj: this I'm saying like this, because if you don't know sequel, this is how usually you speak sequel as well as well
316 00:48:37.610 --> 00:48:42.059 Borivoj: to okay, this for the single joints. Okay? So we have.
317 00:48:42.270 --> 00:49:07.110 Borivoj: If you need to join more than one data frame you, you can use join many. So you join many again, same as any other, many function that they show already square brackets, individual brackets for for different data frames. So here we are joining 3 different data frames, 1st one, then the second, and the 3, rd and the rest is pretty much the same.
318 00:49:08.745 --> 00:49:14.300 Borivoj: Besides this, what is also supported is joining on multiple columns.
319 00:49:14.990 --> 00:49:29.640 Borivoj: Okay, so joining multiple columns for us, for 2 data frames, we again use joining function. But you also can provide in the square brackets, multiple columns 2, 3, 4. However much you want.
320 00:49:29.820 --> 00:49:38.550 Borivoj: Same thing goes with the for the join menu. When you want to join multiple data frames again. Same thing, square brackets. Then you put
321 00:49:38.870 --> 00:49:51.019 Borivoj: individual brackets, and inside you supply multiple joining columns. Very important thing to know is what restriction we have right now here. The restriction right now here is that you cannot
322 00:49:51.180 --> 00:50:06.420 Borivoj: join uneven number of volumes, which means what which means if you wanna join 1st and second data frame on 2 columns, and you want to join second 1st and the 3rd
323 00:50:06.670 --> 00:50:19.930 Borivoj: data frame on one column is not gonna work. So you need to have equal number of columns. Currently, I was, I was unable to implement this. It was failing in testing. So I I currently.
324 00:50:20.040 --> 00:50:24.840 Borivoj: oh, I'm still, I'm still. I still didn't finish this implementation.
325 00:50:25.200 --> 00:50:36.019 Borivoj: But currently, just for you to know, this is a limitation. So if you want to evaluate on multiple columns. You need to have a same number of columns that you are evaluating against.
326 00:50:36.180 --> 00:50:44.949 Borivoj: Okay, so but now we are starting another function for me it was the most logical thing to do when you work on a string columns.
327 00:50:45.270 --> 00:51:00.620 Borivoj: Okay? So we were gonna string columns. My whole idea to create this new function called string functions, which I don't think you have in any other data frame. So maybe this is going to be a little bit weird for you, but for me it was very logical. If you want to work on some
328 00:51:01.120 --> 00:51:20.519 Borivoj: string function or like any function that that of course you have string type. Utf. 8. Or if you come from Json or any other text, virtual whatever, from whatever data source coming and whatever string kind of time it has, you should use string function
329 00:51:21.010 --> 00:51:45.449 Borivoj: because string function is differently. Parsing in the background is differently parsing in the rest of the functions. So if you're gonna place some string function in select, it's gonna fail. It's not gonna work. So make sure if you work. If you are applying some function that is usually working on string functions, make sure to put it in the string function, as you can see just here below select.
330 00:51:47.340 --> 00:52:16.360 Borivoj: So yeah, so we're gonna talk about more string functions just now. So this is more example, as you can see here, join many select string function, aggregation group by all, having many order by many. As I said already, probably 5 times. You can literally write here, select string functions, join aggregation, you can write string, function, select aggregation, join literally. You can write here, however, is most logical for you. It's gonna work the same.
331 00:52:17.410 --> 00:52:24.570 Borivoj: And now, as you can see, 3 fashions have all these stream concant left right. All this most usable.
332 00:52:24.840 --> 00:52:27.849 Borivoj: like most frequently use string functions.
333 00:52:27.850 --> 00:52:31.070 Ray Lutz: Okay? Quick question. When you do joins.
334 00:52:31.660 --> 00:52:38.719 Ray Lutz: can you join without doing the aggregations and selections and create views.
335 00:52:39.630 --> 00:52:44.370 Borivoj: You cannot create views because everything is stored as a data frame in memory.
336 00:52:44.640 --> 00:52:50.629 Borivoj: So each of I don't have lazy evaluation like like polars, has.
337 00:52:50.790 --> 00:52:59.980 Borivoj: for example, porous, has lazy evaluation. So you write right right, and only when you execute, then actually do something here.
338 00:53:00.170 --> 00:53:07.340 Borivoj: There is no such thing as a view. You cannot store ticket as a view, you can only sort thing as a data frame in memory. So
339 00:53:07.340 --> 00:53:09.036 Borivoj: is that part of
340 00:53:09.920 --> 00:53:14.130 Ray Lutz: Apache data, fusion, functionality that they don't have use.
341 00:53:14.990 --> 00:53:31.339 Borivoj: Yeah, they have used. You can use, you can create views. But I didn't implement it. So so the the way I implement this is everything. Come and go through the data, frame everything, come and go through your local memory. So
342 00:53:31.670 --> 00:53:40.230 Borivoj: this is how I implement this. So you cannot create the if. If you're saying view, you want to interview where you want to create view in memory, or you want to create view in.
343 00:53:40.260 --> 00:53:45.579 Ray Lutz: Well, it's just well, but because it's in memory, I mean.
344 00:53:45.760 --> 00:53:58.850 Ray Lutz: this, this thing operates in memory. You're not operating off on a disk somewhere. And that's a limitation of this, right? Because you can't have too big of a data frame. It will run out of memory
345 00:53:59.613 --> 00:54:00.680 Ray Lutz: and then.
346 00:54:00.680 --> 00:54:05.340 Borivoj: Depending on, of of your memory of computer. If you have 32Â GB of memory.
347 00:54:05.340 --> 00:54:15.439 Ray Lutz: So if you have lots of you'd have to have RAM, though it can't be on disk. Right? Yeah. So even SQL lite allows you to have some pretty large
348 00:54:15.690 --> 00:54:18.170 Ray Lutz: tables.
349 00:54:18.410 --> 00:54:25.489 Ray Lutz: and then what they do in those other SQL, maybe it's I, just. I haven't used data fusion before, but
350 00:54:25.600 --> 00:54:30.339 Ray Lutz: they? They let you make views, and then, in order to make
351 00:54:30.900 --> 00:54:41.309 Ray Lutz: you know, then you can. You can join 2 tables. You can join another table. You can join another table, and nothing happens until you select on it.
352 00:54:41.310 --> 00:54:41.810 Borivoj: Yes, yes.
353 00:54:41.810 --> 00:54:42.260 Ray Lutz: Then.
354 00:54:42.260 --> 00:54:43.480 Borivoj: And she's like on it.
355 00:54:43.480 --> 00:54:43.800 Borivoj: Say no.
356 00:54:43.800 --> 00:54:45.950 Ray Lutz: Then it goes back, and it figures out
357 00:54:46.070 --> 00:54:52.889 Ray Lutz: how it's going to, you know, because it has to has to behind the scenes.
358 00:54:53.030 --> 00:55:05.860 Ray Lutz: you know. Take the 1st one, get the key, you know. Find the other one, find the key in that one. Get the data that you're looking for. Pair it up, you know, all the way through. Okay, all right. Thank you. That's my question.
359 00:55:05.860 --> 00:55:07.950 Borivoj: Yes, yes, yes. Yeah. Yeah. So
360 00:55:08.290 --> 00:55:29.849 Borivoj: it is what it is. Currently, it's in, built for in memory data format without lazy evaluation. So when you run it. How you construct your query at the moment you evaluate is gonna start crunching your data. There is some optimization for compressing bytes and everything. But you cannot go
361 00:55:29.940 --> 00:55:43.339 Borivoj: over your limits of your local machine. That's impossible, because this is, for in memory data formats like, say, similarly, like pandas like, you know, Polars has more advanced lazy evaluation. So similarly, like Pispark.
362 00:55:43.850 --> 00:55:56.369 Borivoj: it's lazy, validated as well. You can write, write, write, write, write, you can store store store, use, use whatever, and then you execute, and then it's figuring out. And then you wait for 5Â min. Basically here.
363 00:55:58.070 --> 00:56:01.029 Borivoj: here it's a little bit different. It's made, it's
364 00:56:01.750 --> 00:56:07.740 Borivoj: quite performant. But you have this limitation of of your own network.
365 00:56:08.000 --> 00:56:14.799 Borivoj: It is what it is. Okay. So currently implement the joints in the left, right, full left, same right, same left and right and the left mark
366 00:56:14.920 --> 00:56:21.070 Borivoj: think covers. Most of the are normal humans working with data
367 00:56:22.301 --> 00:56:29.630 Borivoj: string functions. Okay? So, as I already said, let's just repeat a little bit more and show it a bit more examples. So in string functions
368 00:56:30.050 --> 00:56:38.149 Borivoj: you can, of course, create new columns, non-existing like this, like new as new old customer. You can add string values to the.
369 00:56:38.320 --> 00:56:45.290 Borivoj: to the, to the coins. You can, of course, use all these string functions as as usual.
370 00:56:46.520 --> 00:56:59.159 Borivoj: Here are more examples left, trim, right, trim, upper, lower length, left, right, any, as I already said, not to repeat myself any string functions. You need just feel free to to put it in the string functions.
371 00:57:00.010 --> 00:57:11.119 Borivoj: Yeah, so that's it. More examples here replace and stay the repeat reverse left pad, right, bed, and all this other stuff. Still, I'm working more on string types of dates.
372 00:57:12.200 --> 00:57:22.480 Borivoj: There there is casting there is to chart and stuff like this. But I still think I think I still didn't cover all the time and date functions that data fusion
373 00:57:22.610 --> 00:57:32.439 Borivoj: has natively still need to to go through all of these. But most of the like general things I covered. So these are the currently available string functions
374 00:57:33.020 --> 00:57:38.300 Borivoj: that you can use on the in the string in the string function, in in your query.
375 00:57:38.890 --> 00:58:07.930 Borivoj: okay, so windowing, windowing, I was thinking how to implement this because you already saw there is, mostly this logic. We like filter filter, many order by order, by many things like this. But for windowing it's overly complex to have window many, because now you're not going to be able to figure out for me, at least for writing errors, was a little bit difficult if I have window many, and I have a list of windows
376 00:58:08.210 --> 00:58:22.199 Borivoj: to actually to to designate which one is failing. So for me it was a little bit easier to implement this like this, and I think it's also enough, is it's it's more than enough complex windowing to have it as a single as a single function. So
377 00:58:22.430 --> 00:58:25.150 Borivoj: I implement like this. So you can do.
378 00:58:25.630 --> 00:58:30.950 Borivoj: aggregated window functions, ranking analytical window functions. And you also can do rolling.
379 00:58:31.250 --> 00:58:32.330 Borivoj: So
380 00:58:33.080 --> 00:58:43.230 Borivoj: anything that you need. I think it's covered here, and it's gonna be properly parsed. This is was the the hardest thing to implement for parsing, because it can get pretty quick. Pretty complex.
381 00:58:43.650 --> 00:59:01.399 Borivoj: But yeah, I think I covered the the most cases for for windowing, and you can use, of course, in windowing for aggregations all the as it showed before all the aggregation functions you can use here you here we can see some and average, but you can use all the other that they showed before for aggregations as well.
382 00:59:02.650 --> 00:59:05.149 Borivoj: Yeah, that's that's it for that.
383 00:59:05.450 --> 00:59:24.810 Borivoj: Okay, append append many. So I also added, this was not necessary. This is literally like Union, and like Union all, and union all, many, literally the like same functionalities. But I know some people like append and can remember, is it append? So you can use this one as well? It's gonna keep all the rows. It's not gonna remove duplicates
384 00:59:24.900 --> 00:59:36.689 Borivoj: in any case. So what I think it's I'm a little bit biased here in Pispark and other data frames you need to. If you want to append. If you want a union. If you want to do any of this.
385 00:59:36.800 --> 00:59:47.059 Borivoj: you need to provide data frame dot function data frame dot function inside data frame. But here I implemented, if you wanna
386 00:59:47.230 --> 00:59:54.510 Borivoj: append multiple data frames, you just literally say, append many. And you just add each data frame you want to append.
387 00:59:54.660 --> 01:00:00.130 Borivoj: So I think this is a bit easier usage than in other data frame libraries out there. So
388 01:00:00.490 --> 01:00:23.779 Borivoj: yeah, you can see, example here, 5 Json files, you create data frames you do some whatever. This is just simple, like queries. You evaluate all of these. As I said, if you want to use it further, you evaluate through illusion. You need to do this, and then you can use it, append, append many and all these other functions that are coming right now you're gonna see
389 01:00:24.170 --> 01:00:32.420 Borivoj: Union Union all accept and intersect. So Union compiles the rows for both and removes duplicates. Union. All, as I said.
390 01:00:32.650 --> 01:00:37.260 Borivoj: similar like appendment is, gonna keep all the rows, not removing duplicates
391 01:00:37.670 --> 01:00:54.989 Borivoj: except an intersect difference between 2 sets and intersection of 2 sets as well. So this is a small example of of union. So you have 2 data frame, df, one df, 2, and then you evaluate through illusion. And then you union these 2, these 2 tables
392 01:00:55.590 --> 01:01:02.939 Borivoj: and same thing is for union all except and intersect you. You provide you provide this tool.
393 01:01:03.470 --> 01:01:06.629 Ray Lutz: So do the columns have to be exactly the same.
394 01:01:06.630 --> 01:01:14.490 Borivoj: Yes, need to be the same need to have same number of columns need to have same data. Types need to have same call names.
395 01:01:15.660 --> 01:01:16.270 Ray Lutz: Thank you.
396 01:01:18.860 --> 01:01:21.820 Borivoj: Thank you for this question, because I will forget to say this.
397 01:01:23.001 --> 01:01:31.429 Borivoj: So, okay, so union, many union, all, many same same logic. So you create 5 different data frames union many.
398 01:01:31.690 --> 01:01:36.850 Borivoj: You provide us the 1st data frame, because on the 1st data frame, this is very peak
399 01:01:37.000 --> 01:01:39.900 Borivoj: on the on the data. And I this how I establish
400 01:01:40.250 --> 01:01:54.699 Borivoj: after the evaluation, I establish the again, I need to establish again in this case the schema, and then against this 1st file that you provide 1st data frame that you provide. Then I'm comparing all the the rest. Right? So
401 01:01:55.490 --> 01:01:59.377 Borivoj: yeah, so union many and union all many. It's
402 01:02:00.040 --> 01:02:13.030 Borivoj: yeah. So it's you can see union. Many same thing like union, just with many. It remove duplicates and union all, many is same like union. All just you can provide multiple data frames is gonna keep the duplicates.
403 01:02:15.220 --> 01:02:21.309 Borivoj: the pivoting. Okay, pivoting was challenging, very challenging and pivot and on pivot
404 01:02:21.470 --> 01:02:29.120 Borivoj: because are not supported by data, by data fusion. Natively. So I need to
405 01:02:29.320 --> 01:02:40.570 Borivoj: create pretty interesting break around this thing. So a very important thing. This is a sequence function, it cannot be changed with other functions.
406 01:02:40.950 --> 01:02:47.260 Borivoj: So just remember this, you cannot write select target string and then
407 01:02:47.870 --> 01:02:55.489 Borivoj: change. Pivot is not gonna work, pivot and pivot are separate functions from other functions. So you need to evaluate your data frame
408 01:02:56.100 --> 01:02:58.110 Borivoj: to work this. So you're gonna
409 01:02:58.420 --> 01:03:01.089 Borivoj: let's say you wanna people directly on the
410 01:03:01.350 --> 01:03:11.599 Borivoj: row. Csv, you create custom data frame, and you build on the data frame or another version, you work your chaining, you write your query, and then
411 01:03:11.850 --> 01:03:15.570 Borivoj: you evaluate with illusion, and then you can use
412 01:03:15.690 --> 01:03:30.049 Borivoj: pivoting. So you you need to provide illusion. Then, separately, on this data frame, you need to pivot or unpivot. So this is just an example. Here on the screen, CC. Files little bit of data crunching. You use illusion.
413 01:03:30.430 --> 01:03:35.779 Borivoj: This display is not necessary. If you want to see what or or not, you can remove this, but then on this
414 01:03:36.230 --> 01:03:41.480 Borivoj: scalar results scalar rest. You can peel with. Now. Okay, it's very important
415 01:03:42.200 --> 01:03:50.870 Borivoj: to to know this part. You cannot change. You cannot add Pyota above, let's say, after a filter, you, you cannot do this, so you need to be used separate.
416 01:03:53.640 --> 01:04:13.809 Borivoj: Okay, I don't. I don't want to go in the story about futures. But when you work with a sequence, runtimes and sequence functions, just as I said at the beginning, use await. And the question mark, just remember this. You don't need to think about this too much. Just await dot await question. Mark, dot await question, mark. That's all you need to remember from the rest.
417 01:04:14.140 --> 01:04:16.010 Borivoj: You don't need to worry about this.
418 01:04:17.090 --> 01:04:20.990 Borivoj: Okay, people. Same story, identical story.
419 01:04:21.460 --> 01:04:30.740 Borivoj: You can work directly on the on the data frame or after the illusion evaluation you can. You can. You can just call your compute.
420 01:04:32.610 --> 01:04:34.749 Borivoj: Okay, statistical functions.
421 01:04:34.930 --> 01:04:45.189 Borivoj: statistical functions I provide, maybe are going to do more. But for me, whenever we did, it's very important to quick evaluate the data frame to see how many nulls I have, how many rows like
422 01:04:45.210 --> 01:05:06.119 Borivoj: like just checking the data a little bit. So I create 3 different statistical functions. So here you can see on the data frame. So you get some metrics for the for the columns you have. How many records have coming on? All values mean standard deviation and mean. And Max range just a little bit that you have some, you know.
423 01:05:07.272 --> 01:05:15.330 Borivoj: Idea about your columns, what what is going on. So here you're gonna get a report for each individual column. I didn't put all the
424 01:05:15.450 --> 01:05:24.519 Borivoj: all the volumes. It will be too big picture. But below this, you're gonna get 2 more, 2 more results, right for each for each of these.
425 01:05:24.880 --> 01:05:33.689 Borivoj: for each of these columns in your data frame. Of course, if you wanna for example, create new analysis, which I usually do, it's kind of a
426 01:05:33.930 --> 01:05:57.357 Borivoj: thing to do to see how many nulls you have in data frame one of the things to check your data so you can provide none in rust. You have this option type. It's like to think about this like, let's say, is it or is it not I? I can use it, or I don't need to use it. This is option. Right in the background is a little bit more complex. It's
427 01:05:57.910 --> 01:05:59.569 Borivoj: actually helping you.
428 01:06:00.033 --> 01:06:02.020 Borivoj: Not to get the null if you
429 01:06:02.370 --> 01:06:12.049 Borivoj: checking some value, and if he's gonna come now, just to cover this, not to get these problems with with errors. But just think of this like a option like the thing
430 01:06:12.510 --> 01:06:31.320 Borivoj: I can use it. I don't need to use it in this case. You don't need to use it, and what you don't need to use is to specify the column name if you just like, say, data frame any data frame that you created and display new analysis, none is gonna evaluate all of your columns. If you provide the column name in the in the
431 01:06:31.970 --> 01:06:41.449 Borivoj: I call this in the braces. How I say, then, you're gonna validate on the on the on the quality. But if you want to just place none, it's gonna validate the whole data frame. Usually you'd use this.
432 01:06:41.890 --> 01:06:57.869 Borivoj: But anyways, okay, so display correlation matrix for some people that I don't usually use this, but if if you have any need to to see correlation between the the columns in your data frame, you can supply the columns.
433 01:06:58.310 --> 01:07:02.930 Borivoj: And then you're gonna get correlation between between the columns. Usually, I think this is used
434 01:07:04.000 --> 01:07:07.970 Borivoj: from data scientists. They usually do this. I don't usually do this. But
435 01:07:09.050 --> 01:07:11.620 Borivoj: I implement this just in case if anybody needs it.
436 01:07:13.150 --> 01:07:41.389 Borivoj: Okay? So we come now to the Odpc open database connectivity and talking about Mysql and postgres which I implemented here. So for Mysql. If anybody use work with Odbc connections before, you need to know that you need to have a driver without a driver. It didn't work. It's not gonna work so you need to have locally your driver installed for your my SQL, so currently, for example, I use this 9.1 unico driver.
437 01:07:41.800 --> 01:07:51.300 Borivoj: and then it's very important for you to remember. You need to supply these keywords as they are right here on the screen. Because SQL is looking for this
438 01:07:51.410 --> 01:07:54.139 Borivoj: keywords for postgres is different
439 01:07:54.310 --> 01:08:02.810 Borivoj: keywords. You're gonna see now. So basically, this is in multiple rows. But this is just like one string I just as you can see, I slashes just to have. I can
440 01:08:03.290 --> 01:08:07.700 Borivoj: see it nicer. But you can just play this in in one line as well, of course.
441 01:08:08.700 --> 01:08:29.370 Borivoj: And then, as I said before, try to push as much as possible pressure on the SQL. Server. So don't put too much pressure on on data frame execution, because you want to get from the I don't know you can. I test this on 1.6 million rows of data with 32 columns with like with 2 tables.
442 01:08:29.470 --> 01:08:31.579 Borivoj: So you have very nice performance.
443 01:08:31.689 --> 01:08:45.609 Borivoj: But try to try to put pressure on the on the sequel in the SQL engine. So don't put too much pressure. Because if you have multiple millions, multiple tables, you need to join with multiple 1 million rows.
444 01:08:45.950 --> 01:08:54.250 Borivoj: I don't know 5Â min rows each. I don't know. It's gonna be too much for your memory to do this like put crunching as much as you can on the
445 01:08:54.689 --> 01:09:07.310 Borivoj: on the SQL. Engine. So or or limit amount of rows you want to take. If you, you can also keep like limit, 20,000 100,000 rows. It's gonna be
446 01:09:07.420 --> 01:09:23.509 Borivoj: extremely fast without you importing all the data. Once you are happy with what you get. Then you can remove limitation, limit, clause, and just run your query as you want. Just my recommendation. Don't try to break your computer right?
447 01:09:23.850 --> 01:09:24.850 Borivoj: So
448 01:09:25.149 --> 01:09:44.420 Borivoj: once you create the connection string and the query, you just use custom data frame from dB, where you're gonna where you're gonna create the the execution. Then you need to use illusion and give it new alias. As I said in the background, I create the crazy alias
449 01:09:45.340 --> 01:10:01.049 Borivoj: when I, running from dB, but you need to worry about this. Add your alias in illusion, illusion, evaluation. Then, later on you can work with with, you can change your functions, work with the data frame as as usual, as we already saw before.
450 01:10:02.140 --> 01:10:10.700 Borivoj: so similar to Odbc. Remember these names. So driver. Several poor database uid pwd, so this is what
451 01:10:10.830 --> 01:10:14.989 Borivoj: probably is expecting. It's a little bit different than my my skill.
452 01:10:15.610 --> 01:10:36.480 Borivoj: And then, yeah, so driver is post SQL unicode that I'm using. You're gonna maybe need some. I don't know which version policies have, but just check which driver you need. Install the driver in your computer and then you can work. You can work with your database. And same, just create illusion, evaluation. Give it alias, and then you can work it later on with the data frame. If you need.
453 01:10:36.660 --> 01:10:43.901 Borivoj: Why is this? Why would why would you use this like? If you can run sequel directly on SQL, let's say you want to work with
454 01:10:44.500 --> 01:11:03.590 Borivoj: you have a analyst in your team. You wanna give them the data from the database and then they work in fabric. So they have native Gen, 2 data lake connection with azure. Right? So you take the data from the database. You push data on the
455 01:11:04.400 --> 01:11:12.350 Borivoj: you push the data on the azure, and then they take it in fabric and then work later on whatever. So just this is like, let's say, use case
456 01:11:12.560 --> 01:11:21.630 Borivoj: for for, or if you. If you if you do locally and publish, let's say power Bi reports or tableau, or whatever reports
457 01:11:21.670 --> 01:11:45.749 Borivoj: you crunch the data with a data frame, you give Csv files to your analysts that they create locally. They publish, publish, you know, to to to the cloud the reports, I don't know, like there are different use cases. In. In in expert group we have different clients, some smaller Serbian clients that we have. Like some furniture company, they work like this.
458 01:11:45.750 --> 01:12:04.600 Borivoj: we have a VPN to their computer to to their station there in the company, and we just supply from the database the the files, and locally we are analysts are creating reports and just publishing, you know. So smaller companies need the smaller solutions. Bigger companies need bigger solutions. But
459 01:12:04.930 --> 01:12:08.440 Borivoj: yeah, you can, you can find use case, probably
460 01:12:08.670 --> 01:12:12.530 Borivoj: for yourself. So azure bulb storage. We already talk about this.
461 01:12:12.690 --> 01:12:21.030 Borivoj: So we don't need to talk too much. So reading from the azure blob storage currently support the Csv. Json. Still, I need to implement the reading packet.
462 01:12:21.740 --> 01:12:34.269 Borivoj: So it's opposite way. I can read Json and Csv, and then I can store parquet. But I cannot read parquet. Okay, so currently. But I gonna implement reading parquet and storing Json and Csv.
463 01:12:34.480 --> 01:12:42.680 Borivoj: I gonna do vice versa as well. So as I already showed you very important thing. If you didn't work with azure blob storage before.
464 01:12:43.070 --> 01:13:09.479 Borivoj: just check like URL to your container, if it has the word blob that's gonna run on the SDK azure SDK, which is not performant if you're gonna if you have word, Dfs is gonna also work through. SDK, but it's gonna be much faster, because using Delta Lake storage second generation. So it's much, much faster. So ask your
465 01:13:09.650 --> 01:13:20.670 Borivoj: principal, or whoever you are, who is your manager? Just okay. Please give me Dfs code. Right? So dfs, URL, so it's much, much faster execution and reading and everything.
466 01:13:21.290 --> 01:13:28.629 Borivoj: Okay, so you provide your URL, your saas token, and then use from azure with sas token
467 01:13:28.640 --> 01:13:30.740 Borivoj: very bluntly to
468 01:13:31.450 --> 01:13:59.139 Borivoj: function name, but from azure with us token you use to create custom data frame. Again, you can use filtering you, you give it alias. Here I just give alias data. And then later on, you, you write your query, and you work with your data if you need to put back your usually. So, for example, we have one company from from Dubai Emirates. So we take the data from in Json format and we store it back in parquet. For example, this is used case
469 01:13:59.140 --> 01:14:03.669 Borivoj: so you can take the Json files which are like we you have like
470 01:14:03.700 --> 01:14:06.360 Borivoj: I don't know over 500 Json files.
471 01:14:06.580 --> 01:14:18.510 Borivoj: We take it we do some aggregation calculations. We pack it in a pocket and we push it back to the, to the azure. So this one use case that your real life use case as well.
472 01:14:20.510 --> 01:14:47.289 Borivoj: Okay. Rest. Api, okay. So rest. Api. I tried to put as much as possible things I could the most like flexible options I could think of. So you can write from Api. Just simply, this is the most simple use case. You just give the URL, and you just give it the path. So all the from Apis. All rest. Api implementation is taking directly from the Api data and storing in Json
473 01:14:47.550 --> 01:14:59.120 Borivoj: in your local local files. Right? So you just provide the URL, and you provide the path and the name in. I just made this before. It was in the 1st implementation was separately
474 01:14:59.400 --> 01:15:11.520 Borivoj: path and the the file name. Now, I just, I think it's easier to use like this where you just remove the this argument. So you use 2 arguments. So yeah, this is from Api
475 01:15:11.740 --> 01:15:17.549 Borivoj: and then from Api with headers. So you can just create a hash map. So hash map
476 01:15:19.593 --> 01:15:21.286 Borivoj: in order to be
477 01:15:21.880 --> 01:15:50.209 Borivoj: program languages. You have different data structures. Right? Hash map is one of the structures you can use without going too much into the hash map. You just basically storing different values before you're going to use it here. So if you're going to create multiple headers so you can create hash map which you're going to use later on in from Api with headers function so as you can see here in example, one you create, you create mutable
478 01:15:50.960 --> 01:15:51.530 Borivoj: headers
479 01:15:52.070 --> 01:16:17.790 Borivoj: variable? Because why is mutable because you are adding more stuff to it. So it needs to be mutable. You are changing the structure right? Rust is very strict of this, of of types, because rust natively, when you create a variable, it's immutable by default. So if you want to create mutable, variable, you need to put this word mut so moot.
480 01:16:17.910 --> 01:16:28.230 Borivoj: So now you can, you can change. You can change this, this this variable and store store multiple things in in a sequence, in in the variable. So
481 01:16:28.380 --> 01:16:55.670 Borivoj: this is simple example with a single header which so now doesn't make much sense right? But it's going to make more sense. In the second example. But you have 2 headers now. You can store. You can see you can store multiple headers, and then you can use it from Api with headers, and it goes on, I don't want to bore you with this, but you can check Github repo. You have much more example. So parameters also hash map. You can hash map, you insert parameters and use parameters.
482 01:16:55.740 --> 01:17:25.160 Borivoj: Then you can mix with parameters and with headers. So now you have parameters. You have headers, and you supply this to the to the from Api with parameters and headers. So I try to create as much as possible. This is from just if you need dates, you can use this one with dates. You evaluate some data from date range. If you need like, for one year data or something. You can use this one also. You can use this with with parameters as well, of course. But I just create just, you know, if you need something
483 01:17:25.640 --> 01:17:50.380 Borivoj: more specific pagination. So you can grab data with different number of pages. And how many roles per page you want, you want to grab depending, if, like, if data is huge or something. So you can page paginate the data and and then evaluate to the data frame with sort. You can also sort like before even you using the the data frame Api from the illusion, you can sort the data before you receive the data
484 01:17:51.530 --> 01:17:56.579 Borivoj: with headers and source. So headers and sort and and so on. Okay.
485 01:17:56.990 --> 01:17:59.770 Borivoj: now, this is the last, the most
486 01:18:00.371 --> 01:18:06.880 Borivoj: relaxing feature I implemented. This is the my favorite feature, because this is the easiest thing to implement
487 01:18:06.910 --> 01:18:35.679 Borivoj: in a sense that it's you know, it's the most creative thing here. It's not so much technical. It's a lot of Javascript. So Javascript is very flexible. So here we have ability for reporting and dashboarding. So you have, you can create plots, and you can create tables from today last version, latest and the greatest. So these are currently available plots.
488 01:18:36.230 --> 01:19:05.029 Borivoj: Here is an example. So plot, line, plot, plot, time series. I don't want to mention all of this. You can check on the on the Github repo, and then X and y axis you. You give it the true for line. If you want to have these dots and see you're gonna check the I don't wanna talk too much. It's super simple. So it's natively baked into the data frame. So after you create data frame, if you want to create a report, you can create your plots, you're gonna see now. And you also, you can create your tables.
489 01:19:05.570 --> 01:19:09.830 Borivoj: Okay, this is more example of plus histogram box plus character.
490 01:19:09.970 --> 01:19:30.719 Borivoj: And then you can create tables. So you you create some query. So now you want to take certain columns from the query, and you want to order it, or whatever, and you need to pass it through illusion to to create the, to create the alias again, just for evaluation. You need to do this, and then it's ready to be used in dashboard.
491 01:19:31.310 --> 01:19:45.570 Borivoj: So you create list of plots that you create. For example, if you create line Time series, you, you pass this to the to the list. Same thing for the tables. You pass all the tables that you created into the tables, and then
492 01:19:45.800 --> 01:20:02.849 Borivoj: for the report creation, you create report layout and table options. How you wanna have your layout for the report. 2 columns, 3 columns, one column grid gaps, and everything. You can check on documentation there as well, and the table options.
493 01:20:02.880 --> 01:20:25.089 Borivoj: You want to budget tables, you wanna what kind of tables you want, etc? You can all the options you can. You can check there and then you can edit Lg, team. Alpine is the. I think I like this one the most. Keep this one because it have green green lights like our chameleon. So I, using this one ag ag team Alpine.
494 01:20:25.370 --> 01:20:39.040 Borivoj: So it's have very nice and modern look, you're gonna see now and then, as a final step you use create report. So you adding the plots, you adding the tables report title, and where you're going to create your HTML file.
495 01:20:39.240 --> 01:20:41.299 Borivoj: So now the whole point is.
496 01:20:41.360 --> 01:20:55.649 Borivoj: you are creating report into the HTML file which you can share with anybody with your company. Of course, bigger solutions have much more advanced this like tableau power bi whatever. But I wanted to create this
497 01:20:55.680 --> 01:21:24.870 Borivoj: full circle of taking the data, crunching the data, creating the report. This was very important for me, because this was my dream for a long time now. So I want to create some full implementation of this whole circle, as you can see here, where is the sum for python users and other programming language users? It's again optional. You can keep it none. If you don't have plots. You have just tables, keep none for plots.
498 01:21:25.120 --> 01:21:39.159 Borivoj: and vice versa. So everything where you see some this is optional. You do need to use it. Same goes for layout, config and table options. I create default structures for the report. So if you want to use default structures.
499 01:21:39.290 --> 01:21:54.959 Borivoj: you just keep this as none NONE, and it's gonna create for you. And now we're gonna show you the demo how the report looks like. So this is the tables. I this is ag grid table, so you can sort it. You can reorder the columns. You can just
500 01:21:55.130 --> 01:22:06.042 Borivoj: quick like you can make it easier to look. And and whatever you want to do, you can, of course, filter. You can choose the dates and stuff like this for the plus. Of course, you have this
501 01:22:06.990 --> 01:22:10.340 Borivoj: different information you can see from you can zoom in, zoom out
502 01:22:11.120 --> 01:22:22.209 Borivoj: stuff like this. You can slide on the timeline. This is for the time series, if you need. And then most important thing for the users of the business is export to excel button.
503 01:22:22.350 --> 01:22:28.749 Borivoj: Of course, now we have export to excel, and you can export the tables. Of course you cannot export the
504 01:22:29.020 --> 01:22:34.219 Borivoj: the, the plots. You can export everything which you placed in the tables down below.
505 01:22:35.300 --> 01:22:38.059 Borivoj: of course. Okay. So that that's that's about it.
506 01:22:38.920 --> 01:22:59.839 Ray Lutz: Hey? So I got some questions. I love your dashboard thing that is really tremendous. I think that's that's a real selling point. Now, when you're reading Csv, it normally comes in as as strings. Right? So how do you get the data types? Is there some functionality where it tries to guess what they are.
507 01:23:00.190 --> 01:23:13.299 Borivoj: Yes, Arrow, I'm using heavily. Apache Arrow, my best friend in the whole world, Apache Arrow. So I'm peeking into the the I'm peeking, as I said, in 100 rows of the of the data.
508 01:23:13.750 --> 01:23:17.830 Borivoj: or 100 lines depending what it is, or if a Json couple of objects
509 01:23:18.090 --> 01:23:23.359 Borivoj: and then I'm comparing against all the possible existing arrow types.
510 01:23:23.680 --> 01:23:34.980 Borivoj: And then arrow does this actually actually for us? So it's recognized the data type. And it's stored if it's not able to recognize. It's automatically goes to Utf 8,
511 01:23:35.470 --> 01:23:52.970 Borivoj: meaning from Utfa to string. If I'm for example, there is some like very complex date formats which currently I was. I was not previously. Recently. I was working with some pos data from the cash register and the dates on that is like 0. 1
512 01:23:53.550 --> 01:24:09.450 Borivoj: name in the in the string, then the and you have the the here. Then you have this time zones, then you have. You know all this, so it's kind of hard to evaluate all the data types. Right? But 95% of data types are covered.
513 01:24:09.450 --> 01:24:17.439 Ray Lutz: Mostly dates. But what can you override? Is there an override to set the data types? Because since Russ is very picky about this.
514 01:24:17.690 --> 01:24:28.630 Borivoj: No, it's not. That's not. It's stuck like this because I didn't. Wanna as I said already, this should be as much user friendly as possible. But trust me.
515 01:24:28.930 --> 01:24:33.319 Borivoj: 95% of data types are covered. 5% goes to string.
516 01:24:33.650 --> 01:24:48.880 Borivoj: And then later on, if you need to cast, you can use string functions. You can like, you know, you can take the parts out. You can cast to date or whatever you need, so you can. You can work your way around. If there's some complex string that I cannot, I cannot.
517 01:24:48.880 --> 01:24:57.439 Ray Lutz: Is it fair to say that the the main impetus for this is because rust really doesn't have a good data frame package.
518 01:24:58.310 --> 01:25:05.710 Borivoj: Rust has good data frame packages. As I said, polar has very good data frame package, but
519 01:25:05.870 --> 01:25:14.950 Borivoj: user unfriendly, in my opinion. And the reason why they switch 100% to python Api is because
520 01:25:15.190 --> 01:25:19.110 Borivoj: even python, even rust users are having hard time to use it.
521 01:25:19.490 --> 01:25:24.190 Borivoj: So that was the question, is there any? I think I.
522 01:25:24.190 --> 01:25:28.170 Ray Lutz: Because, of course, Pandas has one.
523 01:25:28.330 --> 01:25:33.860 Ray Lutz: This is not the Pandas Api, and that's the more traditional like python based
524 01:25:34.473 --> 01:25:38.809 Ray Lutz: or if you're using numpy, you know. Then they have data frames. There.
525 01:25:38.810 --> 01:25:39.280 Borivoj: Yeah.
526 01:25:39.870 --> 01:25:45.949 Ray Lutz: And they're more of a pythonic kind of, you know. Square bracket.
527 01:25:45.950 --> 01:25:46.370 Borivoj: Yes.
528 01:25:46.370 --> 01:25:49.479 Ray Lutz: Know row column, whatever kind of thing where you would.
529 01:25:49.480 --> 01:25:52.589 Borivoj: Yes, indexing everything here. You don't have indexing stuff like this.
530 01:25:52.590 --> 01:26:07.706 Ray Lutz: Kind of like ours. Data frames, too, where you you say row and column. And then, you know, there's a group by and stuff. Okay? I, yeah, I love this. So in the dashboard, though, that's implemented as is that a Javascript?
531 01:26:08.645 --> 01:26:09.240 Borivoj: Yeah.
532 01:26:09.490 --> 01:26:17.649 Ray Lutz: Does that like? If you create a report, are are the is the report still active like this? Or.
533 01:26:17.650 --> 01:26:26.249 Borivoj: Yes, it's interactive. It's HTML. When you run xtml, you close it, you open it, you refresh it. Whatever you do. It's always active like this.
534 01:26:26.430 --> 01:26:32.059 Ray Lutz: Is there a way to get just the dashboard part separate from the rest of the stuff? I mean? Is it cause that seemed.
535 01:26:32.060 --> 01:26:39.869 Borivoj: I remove. I just removed the Png because I thought it's not necessary. But if you think, for example, it would be.
536 01:26:39.870 --> 01:26:43.750 Ray Lutz: Know what I let's say
537 01:26:44.710 --> 01:26:53.990 Ray Lutz: you, because it seems like a separate layer, if you will, of your solution where it's not specific to
538 01:26:54.110 --> 01:26:58.990 Ray Lutz: the data frame syntaxing or anything else.
539 01:26:59.100 --> 01:27:07.370 Ray Lutz: and would be very useful with just about any other data frame, for example, or anything else
540 01:27:07.500 --> 01:27:12.939 Ray Lutz: is, did you? Did you like come up with the the dashboard
541 01:27:13.260 --> 01:27:19.710 Ray Lutz: as a custom implementation? Or did you use some prepackaged.
542 01:27:19.710 --> 01:27:47.158 Borivoj: Package. So I use for all our package use and all our community versions. So you're not gonna ever gonna spend the time on this also one thing that is very important. This is open source from the 1st day, and it's free to use and everything I'm implementing. I'm checking the licensing and I'm checking is gonna be costly in the future. So all the packages from Javascript for
543 01:27:47.650 --> 01:27:49.530 Borivoj: for excel is Ajax
544 01:27:50.080 --> 01:28:13.939 Borivoj: is Ajax. The library for the excelling and Csv for the for the Ag. Grid is agrid community license or community library. So it's free to use. So this is what I'm using so ag grid for the tables, Ajax for the exporting and the plotly community version, not the the paid one that goes in the dash is used for the, for the plotting.
545 01:28:14.390 --> 01:28:15.380 Borivoj: so everything.
546 01:28:15.865 --> 01:28:20.359 Ray Lutz: Dash, it's mostly plotly dash is what you're using for this.
547 01:28:20.360 --> 01:28:23.160 Borivoj: Not dash, not dash, just plot, link, no plus
548 01:28:23.160 --> 01:28:29.542 Borivoj: and dash are yeah separate thing. I'm not using dash, because for dash, dash is using.
549 01:28:30.570 --> 01:28:37.109 Borivoj: they have the, if I remember correctly, they have us like, think of a
550 01:28:37.690 --> 01:28:46.030 Borivoj: the functions that you need to call the function on the action to be, and to re crunch the data every time I think for that.
551 01:28:46.030 --> 01:28:46.630 Ray Lutz: No worries.
552 01:28:46.630 --> 01:28:58.360 Borivoj: Yeah. So this is, you need to consult. You will need to have constant connection with your data and with your queries once you create HTML, you are totally detached
553 01:28:58.730 --> 01:29:14.420 Borivoj: from your queries and from your implementation and from your execution. That is so. This is the reason why I can implement dash community version dash. But then, I need to also figure out how to have constant connection with your data.
554 01:29:14.820 --> 01:29:17.989 Ray Lutz: And have sliders and things like that. Interactive controls.
555 01:29:17.990 --> 01:29:21.189 Borivoj: Yeah. So currently, I can do some
556 01:29:21.430 --> 01:29:47.529 Borivoj: gymnastics to create some functions, to to be able to filter the data in the HTML report. I can probably do this like, here, you can filter. If you can see for the tables, you can filter the data. You can do stuff like this. So, yeah, you can do this for the tables, but it's not connected with the charts. It will be perfect if you can slider this filter that, and everything is interact in this case. Currently, it's not.
557 01:29:47.640 --> 01:29:59.790 Borivoj: But yeah, it's the most recent. I just start doing this like, maybe 10 days ago, 15 days ago, I don't know. So yeah, we'll work on more. We'll work more on this.
558 01:29:59.790 --> 01:30:01.310 Ray Lutz: For answering my questions.
559 01:30:01.310 --> 01:30:02.450 Borivoj: No problem.
560 01:30:02.630 --> 01:30:07.220 Borivoj: Okay? So pipeline scheduling for me, this is very important.
561 01:30:08.120 --> 01:30:18.949 Borivoj: okay, this is obviously in memory data frame. This is obviously run in memory. But of course you can create rust image in a docker.
562 01:30:19.210 --> 01:30:27.480 Borivoj: And you can create this more out, like, let's say, automatic. But again, if you have a computer in your company that is not
563 01:30:27.860 --> 01:30:32.030 Borivoj: doing much, or you can run this process through the
564 01:30:32.180 --> 01:30:39.449 Borivoj: on your server through. Let's say shell script or something. You can also automate this like this. But, for example, if you just open
565 01:30:39.610 --> 01:30:59.240 Borivoj: Vs code or any code id that you're using, you can create your functions, your queries, whatever you want to do, you wrap it in a pipeline scheduler, and then you can set the oh, sorry. Then you can set the frequency of job repeating. So it comes from 1Â min to 30 days.
566 01:31:00.080 --> 01:31:01.920 Borivoj: I try to implement
567 01:31:02.627 --> 01:31:08.840 Borivoj: but it's it was failing so far. I tried to implement the exact time when you wanna run the
568 01:31:09.010 --> 01:31:11.769 Borivoj: when you want to run the the job. But
569 01:31:12.240 --> 01:31:19.260 Borivoj: it's based on Utc time. So now I want to cover all the time, zones and everything, and it's
570 01:31:19.520 --> 01:31:28.910 Borivoj: takes a lot a lot of more more things to cover and to think about and to have, you know, internal evaluation of your daytime
571 01:31:29.330 --> 01:31:46.170 Borivoj: your system date time. Then to check which time zone you are then to a lot of stuff needs to be covered for this to work. So I just keep it currently like this. So for example, you run it right now. And it's gonna tell you. Okay, process is finished, give you result. And then you're gonna tell next
572 01:31:46.490 --> 01:31:56.169 Borivoj: job run is that time that mean that second, based on the time that you have on your local machine without knowing what is your time zone? Okay.
573 01:31:56.550 --> 01:32:02.239 Borivoj: so quickly is implemented like this. So you create pipeline schedule new, you create
574 01:32:02.926 --> 01:32:08.610 Borivoj: the how like how many frequent, how frequent! You want Job to be run
575 01:32:08.750 --> 01:32:14.119 Borivoj: in minutes, hours, or days, as you can see here on the right hand side. And then.
576 01:32:14.810 --> 01:32:18.580 Borivoj: okay, this is a little bit more specific. But just remember double pipe.
577 01:32:18.680 --> 01:32:21.509 Borivoj: Just remember this. If you're coming from Python, just
578 01:32:22.010 --> 01:32:36.300 Borivoj: double pipe, Async, and then you've wrap it in the credit brackets. Okay, just try to remember this like this. Don't think too much about this. I couldn't run the process multiple times without this.
579 01:32:36.640 --> 01:32:59.469 Borivoj: But anyways, just you go pipeline schedule, new frequency, battle pipe, async, and acrylic brackets, and everything else is, as you already saw. Here, example is, we are taking the data from azure, from azure blob storage. We do some crunching and we are saving data to park it. This is locally saving. You can also do saving back to azure, for example.
580 01:32:59.850 --> 01:33:12.429 Borivoj: So you run this every day once a day, couple of like, or every 4Â h, or every 6Â h. And it's gonna repeat, it's gonna repeat its job. And this is implemented with talk, your chrome scheduler. So
581 01:33:12.830 --> 01:33:17.660 Borivoj: yeah, it's very, very interesting implementation, how they calculate minutes, hours, and everything.
582 01:33:18.590 --> 01:33:29.140 Borivoj: Okay, development. So we come to the end. I hope I didn't bore you too much. So currently, what I'm working on. So, as I said before, inline parameters for filtering and having
583 01:33:30.491 --> 01:33:41.100 Borivoj: problem. Here is the securities problem here securities problem here. And I tried to implement something, but my audit was failing.
584 01:33:42.370 --> 01:33:50.589 Borivoj: There is different template templating crates that you can use here, there, or you can use formatting, but it's kind of a dangerous.
585 01:33:51.100 --> 01:33:53.890 Borivoj: for you know, for the.
586 01:33:55.060 --> 01:34:11.200 Borivoj: you know. So for the injection sequel injection. But maybe this is unreasonable to think, because if you want to use it locally. Who can? Who can do anything to you? Right you. You use it, you execute it, you close it. But anyways I couldn't. Because I have this github actions that I'm running. Every time I'm
587 01:34:11.480 --> 01:34:19.440 Borivoj: I'm uploading new new version. It's it's failing. So I need to think very thoroughly about this because of security reasons
588 01:34:20.390 --> 01:34:26.799 Borivoj: as well as how you're gonna be able to use it without again knowing too much trust. That's very important. So
589 01:34:27.120 --> 01:34:50.940 Borivoj: try to make it simple and safe. It's kind of challenge. So S, 3 Connector working on it. Date functions, I said, already working still didn't cover all date functions in from data fusion and reading packet from azure blob and writing. I didn't say this here, but, as I mentioned before, writing Json and Csv to azure blob. So these are 5 implementations that I'm
590 01:34:51.520 --> 01:34:54.859 Borivoj: currently working on. Of course.
591 01:34:55.720 --> 01:35:16.009 Borivoj: if anybody has any suggestion, please write to me. Tell me, okay, Bora, I need this for my job. Can you implement this? Give me a couple of days? We'll see what I can do if I can implement something, because I'm using this for my job. I have very spectacle skeptical principle that use C sharp and.net for many years.
592 01:35:16.200 --> 01:35:33.949 Borivoj: So every. It was very hard for me to convince him. Okay, rust is good. Let's use rust. Rust is, you can do Job, you know, with less cost. We can do this locally. We can do that, but they're still skeptical, many people. But if you're going to use this in for your work.
593 01:35:34.400 --> 01:35:55.110 Borivoj: just tell me what you need. If you need to convince your boss. We can talk. We can schedule the chat. I can explain everything how it works. I can show implementations, whatever you need. I'm here for you. I spending a lot of my free time on this without expecting anything in return, because 1st and the foremost, I'm using this library for my work.
594 01:35:55.220 --> 01:36:00.789 Borivoj: And I'm doing this firstly, for myself, okay, and then for everybody else.
595 01:36:01.170 --> 01:36:02.130 Borivoj: So
596 01:36:02.280 --> 01:36:10.440 Borivoj: please don't feel free to contact me, to to ask for any feature or or anything that you need. I'm here to. I'm here to
597 01:36:11.290 --> 01:36:13.460 Borivoj: oblige. Is that the word?
598 01:36:14.220 --> 01:36:28.500 Borivoj: Okay? So thank you. If you are one of these 10,000 people that are using illusion. Thank you for using illusion. Here is the crates. You can check the crates, Github and my Youtube channel, where I'm posting from time to time and not so active on Youtube channel.
599 01:36:28.810 --> 01:36:36.740 Borivoj: But yeah, so contact me, let's chat. I'm always open for for chat and for implementations and for improvements.
600 01:36:37.360 --> 01:36:41.230 Borivoj: And of course I do things wrong. And then I fix it.
601 01:36:41.810 --> 01:36:47.170 Borivoj: Okay, so I'm not, of course, perfect. Nobody's perfect. And this library is not perfect, but
602 01:36:47.390 --> 01:36:52.280 Borivoj: what it does currently, I think it does. Well, so yeah, that's it.
603 01:36:52.450 --> 01:36:54.950 Borivoj: And thank you for your patience.
604 01:36:55.450 --> 01:36:57.510 Borivoj: That would be it.
605 01:36:58.260 --> 01:37:04.560 Gabor Szabo: Well, thank you. Thank you very much. It was a along with an excellent.
606 01:37:04.954 --> 01:37:09.690 Borivoj: Sorry I told you before. I thought oh, 9 o'clock! Oh, my.
607 01:37:09.690 --> 01:37:11.930 Dragana Dragana: Thank you, Bora. Thank you.
608 01:37:11.930 --> 01:37:12.610 Gabor Szabo: Thank you.
609 01:37:12.610 --> 01:37:22.660 Dragana Dragana: Everybody must go. It was very. It was very interesting. Thank you so much. Thank you for. And greetings from Uk. Bye, bye.
610 01:37:22.910 --> 01:37:23.620 lapid: Bye-bye.
611 01:37:24.420 --> 01:37:25.769 Gabor Szabo: Okay, so thank you.
612 01:37:25.770 --> 01:37:26.190 Ray Lutz: Or.
613 01:37:26.190 --> 01:37:28.010 Gabor Szabo: Everyone, and thank you for the questions.
614 01:37:28.420 --> 01:37:35.070 Ray Lutz: Did. Did you do any benchmarking, I mean, how is this an attractive thing for speed? Because I know.
615 01:37:35.070 --> 01:37:55.659 Borivoj: I did. I did a lot of benchmarking, still fastest pocket reader and writer in the world for a single node execution, because I didn't implement this. This has come from data fusion. We have fastest reader and writer in the world. So if you need fast reading and writing with pocket. Okay, I I was, as I said before, I was messing with appending.
616 01:37:55.830 --> 01:38:00.229 Borivoj: So it's it's not so easy. But still performance is.
617 01:38:00.230 --> 01:38:01.040 Ray Lutz: Or pending.
618 01:38:01.210 --> 01:38:08.070 Borivoj: Yeah, so appending is a little bit different. But writing and reading is the fast in the world
619 01:38:08.290 --> 01:38:13.169 Borivoj: for Json grading. It's very, very fast, even for the flattening.
620 01:38:14.440 --> 01:38:20.170 Borivoj: I didn't want my selling point to be. I'm faster than pandas, like some other libraries.
621 01:38:20.780 --> 01:38:24.759 Borivoj: I don't think that's the most important thing. I think the most important thing is accuracy.
622 01:38:25.240 --> 01:38:27.630 Borivoj: It's a good and safety.
623 01:38:27.870 --> 01:38:49.079 Borivoj: Okay? And simplicity of using. Okay? So I'm not in a I'm not in a running track trying to beat anybody by speed. I want to create something that is safe, stable and and good to use. So yeah, I did benchmarking. There is some benchmarking post results from benchmarking. But
624 01:38:49.250 --> 01:38:51.379 Borivoj: yeah, as I said, most important thing.
625 01:38:51.570 --> 01:38:54.160 Ray Lutz: Yeah, I think it's not that hard to beat Panda so. And.
626 01:38:54.160 --> 01:38:58.530 Borivoj: Yes, that's what I want to say. Some people brag. They are faster than pendants.
627 01:38:58.750 --> 01:39:13.329 Borivoj: you know, but I never want to do this. Pandas is my love. I love pandas. I use pandas like I was using pandas a lot before, so I would never trash and bash pandas on speed. Pandas is great.
628 01:39:13.520 --> 01:39:15.679 Borivoj: Go use blunders, go use.
629 01:39:16.085 --> 01:39:20.144 Ray Lutz: Did you? In entertain the idea of using the pandas?
630 01:39:22.155 --> 01:39:23.130 Ray Lutz: Syntax!
631 01:39:24.280 --> 01:39:40.240 Borivoj: As I said the reason I was overwriting these 2 times, it's because the problem is with the data fusion different. Apis. I tried to create very similarly at the beginning. If if I go back to 1st couple of versions, it was very, very similar, like
632 01:39:40.340 --> 01:39:53.689 Borivoj: column dot function, or, you know, aggregations with dots like some mean. But, as I said, I hit the wall for expansion with more complex things.
633 01:39:54.030 --> 01:39:57.690 Borivoj: Then I lean back on the the SQL. SQL. Reference.
634 01:39:57.690 --> 01:39:58.350 Ray Lutz: Hmm.
635 01:39:58.350 --> 01:40:07.970 Borivoj: It is what it is. It's life. You need to make your peace with the limitations and work on what you can work. Otherwise I will need to implement, like polars, everything from the scratch.
636 01:40:08.250 --> 01:40:13.259 Borivoj: all my functions, and then it would take couple of years for this. But.
637 01:40:13.260 --> 01:40:21.320 Ray Lutz: Another thing that's that's come up with, because I've been working on a similar data frame package called Daffodil, and I invite you to take a look
638 01:40:21.510 --> 01:40:23.820 Ray Lutz: within python.
639 01:40:24.060 --> 01:40:29.610 Ray Lutz: But one thing that's come up lately are are embeddings, so they want to put in
640 01:40:30.314 --> 01:40:34.030 Ray Lutz: say, like a numpy array in one cell.
641 01:40:36.760 --> 01:40:39.379 Ray Lutz: You want to have 384 bits.
642 01:40:40.390 --> 01:40:44.950 Ray Lutz: all right, 384, 32 bit floating point numbers.
643 01:40:46.200 --> 01:40:48.810 Ray Lutz: And that's what fits into a
644 01:40:49.050 --> 01:40:53.716 Ray Lutz: cuda like a a you know. Gpu
645 01:40:55.620 --> 01:40:59.829 Ray Lutz: at one time. So and then so the the record
646 01:41:00.010 --> 01:41:10.740 Ray Lutz: is these, these individual embeddings, and this is a word that they use so. And then they really don't want to index into these.
647 01:41:11.170 --> 01:41:11.780 Borivoj: Oh, okay.
648 01:41:11.780 --> 01:41:17.320 Ray Lutz: Put that into one cell. So what do you think the outlook is? Oops?
649 01:41:19.330 --> 01:41:21.250 Ray Lutz: For putting in
650 01:41:23.830 --> 01:41:31.199 Ray Lutz: somehow, like this is what daffodil is good at because it uses python data types. So you can nest whatever you want into a cell.
651 01:41:31.610 --> 01:41:34.040 Borivoj: Tell me you have a front end in what.
652 01:41:34.960 --> 01:41:37.820 Ray Lutz: Pyth daffodil is a python.
653 01:41:37.820 --> 01:41:44.700 Borivoj: So no, okay, but you, you're saying sell. I'm imagining like a spreadsheet of some sort, and you are.
654 01:41:44.700 --> 01:41:49.910 Ray Lutz: Yeah. So in your case, if there's a column and a row in a database, so you have
655 01:41:50.030 --> 01:41:52.800 Ray Lutz: a record and you have a column name.
656 01:41:53.290 --> 01:41:54.310 Borivoj: And that.
657 01:41:54.750 --> 01:42:01.840 Ray Lutz: That's that item normally isn't is a number in there, like in data frames, they think of it as one number.
658 01:42:01.840 --> 01:42:02.250 Borivoj: Yes.
659 01:42:02.250 --> 01:42:08.500 Ray Lutz: What they want to do is put 384, the 32 bit floats.
660 01:42:08.890 --> 01:42:18.689 Ray Lutz: Okay in one cell. So they don't really want to split out and have 32 big, 384 columns, because they never referenced the columns.
661 01:42:19.793 --> 01:42:21.320 Ray Lutz: And so
662 01:42:22.100 --> 01:42:26.370 Borivoj: But suppose that it's tricky, with a floating point in a single set.
663 01:42:26.370 --> 01:42:30.649 Ray Lutz: Now in in postgres, they have a thing called Pg. Vector.
664 01:42:30.870 --> 01:42:31.450 Borivoj: Oh, okay.
665 01:42:31.450 --> 01:42:41.350 Ray Lutz: Which which provides not only that, but functionality in terms of doing nearest neighbor
666 01:42:43.980 --> 01:42:48.810 Ray Lutz: searches. So you can say, I want to look in end. Space. So.
667 01:42:48.810 --> 01:42:49.190 Borivoj: Good.
668 01:42:49.190 --> 01:42:53.240 Ray Lutz: 384 is like a 384 dimensions.
669 01:42:53.430 --> 01:43:00.899 Ray Lutz: and you have a float, me, you know, being where you are, and then each one of these may be many hundreds of thousands of
670 01:43:01.020 --> 01:43:01.980 Ray Lutz: items.
671 01:43:01.980 --> 01:43:02.420 Borivoj: Yes.
672 01:43:02.420 --> 01:43:07.699 Ray Lutz: Are different, you know. I guess settings within a neural network or something right and.
673 01:43:07.700 --> 01:43:08.410 Borivoj: Okay.
674 01:43:09.110 --> 01:43:24.880 Ray Lutz: And so you want to find the nearest neighbor either. And there's usually in in Pg vector and postgres, they have 3 different ways of doing the searching. So that's 1 thing I was coming across lately of a
675 01:43:25.020 --> 01:43:34.400 Ray Lutz: I'll just let you know, in case other people may be interested in that in within your package, that they may be wanting to have these embeddings.
676 01:43:34.930 --> 01:43:39.300 Borivoj: I will. I will need to see. I need to do some research on this.
677 01:43:39.300 --> 01:43:46.650 Ray Lutz: Check out check out postgres in terms of their in terms of
678 01:43:46.650 --> 01:43:49.959 Ray Lutz: their pg, vector which I think is a 3rd party package.
679 01:43:50.580 --> 01:43:53.966 Ray Lutz: But yeah, you've done a lot. And
680 01:43:54.560 --> 01:44:02.749 Ray Lutz: I think that maybe my recommendation would be maybe to skinny down some of the things that you have in your package.
681 01:44:03.790 --> 01:44:13.529 Ray Lutz: so that you don't have so much to maintain and and allow people to implement things such as the Api. I would probably recommend
682 01:44:13.690 --> 01:44:15.829 Ray Lutz: not having that in the package.
683 01:44:16.608 --> 01:44:19.001 Ray Lutz: Because it isn't really
684 01:44:19.730 --> 01:44:24.960 Ray Lutz: Well, maybe you can come up with some sort of a a very.
685 01:44:25.220 --> 01:44:28.999 Ray Lutz: you know. Simplified version of that, or something.
686 01:44:29.000 --> 01:44:43.650 Borivoj: Yeah, I was thinking to split, to split. I was thinking to create couple of versions like 3 point O, 4 point. Oh, I don't know something like this. And to strip to just leave data frame in one version, and maybe, like, you know, in 4 point O, to add
687 01:44:43.810 --> 01:44:48.640 Borivoj: dashboarding in this one to 8 Api, something like this. Because, yeah, it's heavy.
688 01:44:49.050 --> 01:44:51.320 Borivoj: Just to test single thing. It takes.
689 01:44:51.320 --> 01:44:52.500 Ray Lutz: Heavy. Yeah.
690 01:44:52.500 --> 01:44:54.800 Borivoj: 5, 6Â min to compile and then wait.
691 01:44:54.800 --> 01:45:09.770 Ray Lutz: Well, the thing is that also you're doing a lot of you're doing a lot of redoing of the SQL. Syntax, and I would wonder cause I was thinking about a similar extension to Daffodil, and my 1st
692 01:45:11.440 --> 01:45:12.550 Ray Lutz: way of
693 01:45:12.920 --> 01:45:19.254 Ray Lutz: wondering if it would be better is to just allow people to put the the SQL.
694 01:45:20.090 --> 01:45:22.350 Ray Lutz: in in a string right.
695 01:45:22.590 --> 01:45:30.577 Ray Lutz: and let them just run that. Now, once you do that, when what you're doing you may as well just use it directly, but
696 01:45:31.180 --> 01:45:31.900 Borivoj: Yeah.
697 01:45:31.900 --> 01:45:39.890 Ray Lutz: For the daffodil has a syntax that's similar to what Python normally uses, with rows and columns.
698 01:45:39.890 --> 01:45:40.450 Borivoj: Hmm.
699 01:45:40.450 --> 01:45:43.209 Ray Lutz: Indices right? And then those can be.
700 01:45:43.330 --> 01:45:49.710 Ray Lutz: Those can be strings or integers, or they can be lists of strings or lists of ranges, or whatever you want.
701 01:45:49.830 --> 01:45:54.470 Ray Lutz: And then you can select from your data frame
702 01:45:55.950 --> 01:46:05.849 Ray Lutz: and what I was looking at doing was because everything is fine with daffodil and memory. Yeah, I don't need to do any of the.
703 01:46:06.230 --> 01:46:06.660 Borivoj: Yeah, yeah.
704 01:46:06.700 --> 01:46:14.750 Ray Lutz: The the sequel commands, and also the one thing that it lets you do very well is to use python.
705 01:46:15.260 --> 01:46:26.729 Ray Lutz: You know, custom, functionality, to create new things, and that may be something that would be hard to do. I know they have. I was looking in the the documentation for data fusion where they have.
706 01:46:26.980 --> 01:46:29.370 Ray Lutz: You worry. You can create a function.
707 01:46:29.750 --> 01:46:34.199 Ray Lutz: a a custom function, right? And it's for the whole.
708 01:46:34.460 --> 01:46:40.519 Ray Lutz: So it looks like they take a whole data frame. You take one data frame, maybe 2 and create a result.
709 01:46:42.365 --> 01:46:44.200 Ray Lutz: That's yeah. Okay.
710 01:46:44.200 --> 01:46:46.889 Borivoj: It's not you want. So I think.
711 01:46:46.890 --> 01:46:48.681 Ray Lutz: No, I'm just saying that that
712 01:46:49.900 --> 01:46:55.789 Ray Lutz: with with you, if you stay in python data types. So there's a data type conversion issue
713 01:46:56.030 --> 01:46:57.960 Ray Lutz: when you go from
714 01:46:58.610 --> 01:47:05.000 Ray Lutz: from python into pandas. It takes forever. I mean to go from, you know, when you think of it.
715 01:47:05.260 --> 01:47:13.340 Ray Lutz: Basically, unless you're doing 32 call, you know, column or
716 01:47:13.460 --> 01:47:17.530 Ray Lutz: columnar functions like summing that, you know.
717 01:47:17.780 --> 01:47:25.750 Ray Lutz: Deviations, Max. Min. All of that stuff on columns that you want to do, which pandas is really fast doing.
718 01:47:27.980 --> 01:47:35.760 Ray Lutz: But you have to do 32 of them to make it worth changing from python data types into pandas.
719 01:47:35.960 --> 01:47:37.329 Ray Lutz: It takes that long.
720 01:47:37.590 --> 01:47:47.370 Ray Lutz: So what I was doing initially, I was using pandas all the time. I was just using data frames like nothing's flat. And I would. And I found that I was going in and out of them.
721 01:47:47.530 --> 01:47:52.759 Ray Lutz: And this was this was a big hog of my time, because I was time sensitive.
722 01:47:52.880 --> 01:48:01.039 Ray Lutz: And so then I was saying, well, I'm just going to stay within python data types, and it runs much better and easier in many ways and sometimes smaller, too.
723 01:48:02.660 --> 01:48:12.910 Ray Lutz: But it is something that people are used to the syn, the syntax of pandas, you know, and if you go to them, and you say I want you to use a new data frame
724 01:48:13.724 --> 01:48:18.550 Ray Lutz: methodology. And if you say it uses the same syntax as pandas. But it.
725 01:48:18.880 --> 01:48:23.150 Ray Lutz: you know, 10 times faster or whatever problem. Big problem.
726 01:48:23.150 --> 01:48:27.641 Ray Lutz: Great, that's great. You can just pop it right in. And
727 01:48:28.660 --> 01:48:37.739 Ray Lutz: So anyway. So thank you so much. I would like to continue the conversation. I'm doing a presentation, I think. In is it? March?
728 01:48:41.140 --> 01:48:42.300 Ray Lutz: On Daffodil.
729 01:48:43.014 --> 01:48:50.421 Ray Lutz: Come and take a look at that. I really like your dashboarding. I would like to somehow adopt that, and and maybe port that over to Daffodil.
730 01:48:51.774 --> 01:48:57.349 Ray Lutz: Because that's that's pretty cool, and would would be really helpful to
731 01:48:57.807 --> 01:49:05.309 Ray Lutz: and and maybe maybe it is easy if you're using a lot of stuff. That's I just don't know what those packages are that you were using.
732 01:49:05.310 --> 01:49:05.750 Borivoj: Hmm.
733 01:49:05.955 --> 01:49:08.010 Ray Lutz: But I'll take a look at your code and see.
734 01:49:08.010 --> 01:49:19.249 Borivoj: It's very easy. I am like a C plus plus programmer. It's I'm like a header file. It's very easy to go through my code. I'm not a modular guy. So you go. You find one file and you find everything you need
735 01:49:19.920 --> 01:49:24.039 Borivoj: perfect. Hey? Bora, thank you so much for your time appreciate it.
736 01:49:24.040 --> 01:49:29.959 Borivoj: Thank you for all the questions and for engagement. It was very helpful, because some things I will forget if we didn't. If you didn't.
737 01:49:29.960 --> 01:49:33.940 Ray Lutz: I like it better when people ask questions myself, because
738 01:49:34.390 --> 01:49:41.629 Ray Lutz: it's like, Oh, gee! I'm actually bouncing off something other than just names on a wall. Okay, thanks.
739 01:49:41.890 --> 01:49:47.669 Borivoj: Thank you, Ray. Thanks again. Thank you for both of you. It was even more interesting.
740 01:49:48.118 --> 01:49:51.710 Borivoj: It was 3Â h long. But thank you.
741 01:49:51.710 --> 01:49:57.579 Gabor Szabo: Yeah, okay, it's fine. So thank you very much. Everyone. And goodbye.
742 01:49:58.320 --> 01:49:59.360 Borivoj: Hey, Joshua.
