---

-- X DAYS OF CODE --

---

UPDATE (2 MONTHS LATER)

---

I NEED BREAK A BREAK FOR COUPLE OF WEEKS. HOMEWORKS AND MIDTERM EXAMS SEASON IN SCHOOL ARE TAKING SO MUCH TIME!!!

I ENDED MY CHALLANGE BC IM SO BUSY WITH MY MOVING ONE COMPANY TO ANOTHER AND MASTER PROGRAM COURSES!!

---

I made my data structures and algorithms homework after working day with c. You can find it here as week 2.

---

Today There was one of the last lessons of my Data Engineering Bootcamp. We studied about an example with Spark and Kafka. I should build a project with these tools in 3 weeks. I learned some basics about these ones.

---

Today I got job offer as Data Analysis Specialist from "Hepsiburada" - ecommerce company that I mentioned #days52 to days 57-. In the new year, my level of knowledge will increase a lot. My learning process/trend continue to raise sharper.

And yea, today I code again. With my Data Science team, we did some fixes and committed our latest prediction of the test set. So there is nothing to do with project. 2 days later we'll present our process.

---

Friday I solved my Data Structures and Algorithms homework questions with c. Here is the codes. Actually I had no so much experience with c, and lecturer wanted this: "We should solve problems with no math functions. My codes work well.

And in this 3 days we built our model with my Data Science Bootcamp team.

Bootcamp organisation wanted best f1 score from us. Actually only f1 score is not good metric for our success I think. But we tried it.

Btw our f1_score is so bad :D it's about %60-62, but I think we can present it well with a ROC-Curve and our preprocessing and feature engineering part.

We'll see.

---

Today we went a "developers meeting" to a bar with my friend. I meet more than 20 people there. There were awesome guyz :D

---

We started building model part of our Data Science Bootcamp Project. And we did a statistically succesfull approach about feature engineering parts.

We have 24 sequential month columns. Actually this 24 sequential months are started from different month for every observation/rows. The only information we have they are sequential.

In this situation we cant say that Month_1 is same month for every of them.

• For observation_1 month_1 = January, month_2 = Jebruary maybe but, for observation_2 month_1 could be equal to March, and month_2 = April.

So generalization is not possible with all columns. For example when we check correlation, coefficients wouldnt be logical. Because we cant compare same months like this.

• So we decided to take difference of all sequential (with 12month lag) months, so we get 12 difference scores.
• And we calculated some statistics, and standardize them. This approach gave us some logical patterns.
• I liked our approach so much :D

---

Whole weekend I was with my Data Science Bootcamp team. We understood electricity data and decided how we can handle with the problems.

At the same time Sunday there was last lesson of the Data Science Bootcamp (7 week over). And we presented our study to lecturer. We took some advices.

Actually members of my whole team are from different branches. And some of them dodnt know the topic well. So this project weeks are felt me like a lecturer. I explained everysteps, and their minds are little confused.

I wanna follow the steps I decided but we decide together, so some of the steps didnt gone like I wanted. This is a team work, it is good with this types of pieces maybe Idk :D

---

I have a lot of homework in my graduate classes. I did my homework these 2 days.

---

I'm in project week with my Data Science Bootcamp. We'll analyze Theft and Loss Estimation data about electricity.

This week's topic understanding the data and preprocess it before building model steps.

I found a really important data leak on dataset, lol. They'll update the dataset and give it to all teams again.

---

Today I had data engineering bootcamp. We learned/repeated some about spark and hadoop.

And we talked about our projects, I'll built a project with twitter API.

---

Today's topic is technical interview about my project (from day 57) with a huge e-commerce company :D

I feel so happy, bc my interview gone very well. They asked questions about every steps of my project. And I explained every point.

I had some little mistakes and I fixed them in seconds front of them, and I did some live coding on living application.

I thought that I had some mistakes about markov chain and funnel analysis questions. But they said that everything is good, my approach was little different but right and my answers is true they said.

---

I had 5 hours lesson today with Data Science Bootcamp. We studied about Time Series Analysis. Actually this area smth I knew, I refreshed my knowledge.

And I have a important technical meeting tomorrow. So I studied some theorical and applied parts to be ready. So I feel good :d

---

Along with this 2 days I focused 2 topic

First one; I have a meeting about a project that I finished 1 month ago, so I forget most of the details. I studied about the project again. I studied on the parts that others could ask questions.

And I had a hm from my Data Science Bootcamp. I stuied about it. There was some good questions like:

• How can you handle with the classes that not included by training set? or Probability threshold of a classifier how could effect the result? etc.
• Actually there is some available solutions. But I handled with them again and It was informative experience to handle these in a hw.

---

I searched some about creating stream data APIs and I tried some. Actually creating API with web scraping looks not true way.

And I decided create API from my own application. I dont know how but I tried some today. I can watch the logs of my website; I can take every clickstream data -events as data. Actually docker logs show them to me, so It could be possible.

I tried to send them kafka and I think this must be the right way to create an API. If it'll be live-stream this scenario looks ok.

Actually most of my trials was between Kafka-docker-web application and python today but I learned so much today.

---

I searched some concepts and created a basic example of my next Web Scraping blog post.

And I did my homeworks of master/university lessons. (This was not directly about coding but the logic).

---

Today there was Data Engineering Bootcamp and we deep dive Spark and Kafka usage again.

And I talked about my NiFi project little bit. And I took some advices and bad feedbacks. I'll publish my project but I should create better one later again.

---

Day Off. I visited my ex university (statistics department) and talked with my teachers. Today was so good.

And in the end of the day there was a meeting with my blog community (SistersLab).

---

Full day I had Data Science Bootcamp. We started to explore analysing steps, and which little touches we should do.

At the same time we studied about optimization.

And I should share this; I realized big mistake in the course about train_test_split and standardization. Common big mistake faced me again in the course. And I explained true way to handle with this situation, statistician part of me banged hist fist on the table :D

---

I studied again my old data engineering notes. And I updated my NiFi project. I added MongoDB writing processor.

I tried to make my own stream data API, but after that I realize this is not actually live stream, so I didnt add this to project but It was good things to try.

And I realized today, There is a dump part on my twitter boot application. I was reading all website with web scraping but xml parts required different handling strategies, I realized it now. So I'll update repo soon.

I published blog NiFi project blog post here.

---

Day off, I dont count today. I created my good looking CV.

Glad to see that my CV has been starting to fill with good points :D

---

Today after working day there was additional lesson on my Data Science Bootcamp.

Actually there is no special gain but we solved some dataframe manipualation problems. Some of the problems solved by pandas but some of them required some specific functions.

---

Today I had a course on Data Engineering Bootcamp. And we started to explore cloudera system deeply.

How can we manage cluster and Which type of controls we should do on the system (for example we should check YARN for status about the system sources)

And we did some example on zeppelin. How we can read data from HDFS by spark, and connection types between pyspark and sql on zeppelin.

I'll watch this lesson again and study about it soon. There was so much details.

---

I launched a vm on Google Cloud Platform and started a data engineering project.

I installed nifi, kafka and docker on it. And I set nifi mysql connection between them. My application fed by stream data again. I'll share it here after I finish it.

And today I had a lesson with in Data Science Bootcamp. I made live coding with the lecturer.

Content of the lesson was about insurance dataset. This was first time that I took insigt about insurance data. This was one of the lackness of mine but Today I learned some about oversampling and undersampling and the use cases.

---

Today there was no one big piece but many little one.

I wrote latest version of my Web Scraping blog post. And I showed some coding parts in it.

We did a group work assignment with my Data Science Bootcamp team. Before the analysis, we visualized the variables and preprocessed the dataset (with 80 variables).

---

Today I decided latest version of the page that I'll use on my Web Scraping Blog Post.

And I write the coding part in a notebook. In next couple days I'll write the post. Here is the web page I created:

---

I wasn't feeling well today. I didnt do any spesific thing.

---

Today I didnt spend my time effectively enough. My energy was so low.

There was additional course of my Data Science Bootcamp. We took some theorical knowledges about univariate and multivariate outliers / anomaly detections.

I tried some basic steps about web scraping to write my blog post better.

My energy level of today was %35.

---

Today I changed my focusing area little bit. I'll write a blog post for SistersLab, and my next blog series gonna be about web scraping.

Actually scraping data without permission looks little illegal. And showing the method all the people who read blog is also not good without permission.

So I decided to create my own web page to give my example about web scraping.

Today I created a web page on github-pages to scrape. This is only a draft now, I'll whare here it when I finish.

---

Today I discuessed with my senior colleagues about Kafka, Nifi and some other message queue tools, and their use cases/reasons.

And after woking day I connected Kafka with Python. And I sent data from Python to trigger some events on Nifi.

• So I sent data from Python to Kafka as producer
• Connedted Kafka with Nifi
• Kafka took data/messages and processed with Nifi
• And some conditions triggered Slack to send message

---

In these days I studied about Nifi and Kafka. Actually this 2 are not directly coding maybe but they count for a data scientist / engineer I think.

I recreated some projects that allready created, and I took some insight.

Then I took some data from stream data APIs with Nifi and made some ETL processing.

• Used JoltTransformJSON and the queries to select and transform data. (JoltTransformJSON is a good processor but little complicated for some tasks)
• Created some new fetures to trigger some events.
• Triggered Slack and ElasticSearch with the results. (I send message to Slack and wrote data to ElasticSearch)

Additionally

• I connected nifi with Kafka.
• But I'm so fresh with Kafka, so I produced data from only bash of terminal (ubuntu). So there is no datasource yet.

NOTE: I made everything on my local ubuntu machine. Elesticsearch website/account and My slack workspace.

Most of my times passed with trying and failing. I tried this steps with more than 5 stream data APIs (exchanges, flighs, crypto currencies etc)

---

After working day I studied about HDFS on big data cluster. I made so much mistake when I run the command and couple times I was lost in the system. But Ok. After practicing I feel more confortable with the opeartions in file system. And sandbox also helped me a lot to check my operations.

And I finished the homework of my Data Science Bootcamp. There was a question about "Effect size". Oh my old friend Meta Analysis.

---

Today I had a lesson on Data Engineering Bootcamp. And today we learn some about HDFS.

We connected a cluster and managed it with sandbox. At the same time mostly we practice on terminal.

We sent some tables from mysql to cluster. And we study on some tricks and rules.

Actually this is smth little hard to understand. So I'l practice with it couple days more. At the same time it's so cool!!

NOTE: Btw Im on 2 bootcamp about 2 weeks. Data Engineering Bootcample started 4 months ago, and Data Science bootcamp started 3 weeks ago. Data Science bootcamp is on weekend (2 additional lesson at weekdays) and Data Engineering one at Tuesdays and Thursdays.

---

Today after work I study about my Data Science Bootcamp tasks. I had some homeworks (generally easy things).

And I studied on some data structures and algorithm chapters.

---

Today I finished my project, later deployed it on cloud linux machine and watched logs to check are there any problem. And All done.

This week has been very tiring. Time to evaluate myself / process:

Sample pieces of the app:

---

Today I got an email from AWS, I selected as AWS Community Builder, I have 15 days to decide if I wanna contunie or not. This is really cool.

And today is weekend, holiday? No. I had things to do

I finished markov chain part, and implemented it to web app, everything is clear.

Business Report task is done. I added all the comments and the tables to web app. No problem, And I fixed some mistakes.

KPI parts mostly solved, Almost no any problem.

As a summary of my 2/5 days my process is smth like this:

• Descriptive statistics task: 95% finished
• I should only add title for the plots, they will be dynamic but its ok, I can solve it less than 1 hour.
• Business report task: 100% finished
• Its ok. Np.
• Funnel Analysis: 50%
• I solved markov chain problem and implemented it to webb app.
• Tomorrow is the last day and the day that I handle with the funnel analysis.

---

Today after working day I continue to implement KPI task to web app. Visually everything is ok now.

• But Details are so much you know. There is some problems with the comment-plot couples.

I stated handle with markov chain question. Actually I solved most of it but There is some questions on my mind, Some of the consepts are not clear for me yet.

I imlemented Business Report questions to web app.

• I took screenshots of the codes, Im showing them with the comments.
• But I dont keep them in folder/local. I upload them to github-issues, so github created a link for me. This is little tricky and awesome thing :D

As a summary of my 3/5 days my process is smth like this:

• Descriptive statistics task: 70% finished
• There is so much detailed things to do. Actually I spend so much time with this but there is more details to do.
• All the tables created, comments and plots-dropdowns connections should fix, there is some errors.
• Business report task: 85% finished
• Screenshots of the codes added, most of the comments added.
• Results'll implement to dashboard/dash app
• Funnel Analysis: 30% finished
• Markov chain partly mostly solved, But this part should be added to web app, I didnt started this yet.
• I searched some notebooks about funnel analysis, but this task seems little hard for this week :/

---

Today after working day continueing was so tiring for me

I created a layout for KPIs questions (showing descriptive statistics).

For each entity I creted a cardbody and in this card there is some tools (like dropdowns and date pickers) to specify visuals. and I added comments for each possible situation vor each question. So there are about 3*8 = 24 paragraph comment about visuals

I should create summary statistics tables for entities but day ended so quickly, so maybe tomorrow I can solve this.

As a summary of my 2/5 days my process is smth like this:

• Descriptive statistics task: 50% finished
• summary tables'll be added
• for last 3 entity dynamic visuals'll be created
• Business report task: 65% finished
• Some repetitive tasks'll convert functions
• Results'll implement to dashboard/dash app
• Funnel Analysis: not started yet
• I read some articles about funnel analysis, so I have some insight about the task. I think I'll handle with this task well.

NOTE: I missed the service bus in the morning, I couldn't wake up :D

---

After working day I solved almost all 8 questions well, but there are some details to fix. Next step is gonna create dynamic dashboard with them.

There are about 8 complex question about multiple datasets in test cases. For the solutions I should make them join. And I think true way to solve them is writing queries in SQL, but actaully I'll create a dashboard with dash end of the tasks. And I think it is ok to writing the queries in python (but I know this could not best option).

• I chose not to use SQL bc, I'll publish my app with docker on linux cloud machine. And I have some problem to create SQL on docker and manage it. Yea I know mysql could be good option on docker but as I said, this could make my process harder and this is a task that time consuming for me in this situation.

There is a question about funnel analysis on the test cases, and I heared funnel analysis first time in this cases. So I searched about it and read some articles, blog post about the idea behind it and the applying steps.

And I searched and tried to understand some github repos / projects about funnel analysis, I think it seems ok to solve the problem.

NOTE: Actaully I remember something like "funnel plot" from my graduation project of Univerty years. It is about Meta Analysis, and the scope of the data.

---

I got some test questions of a e-commerce company today. And I set a deadline of 5 days for tasks.

Actually I didnt study on e-commerce data before, but I think I can learn and handle with them well.

One of the cons is I work full time in first 3 days. (last 2 days are weekend)

Till end of the day I tried to understand cases and the problems that they wantted from me to solve. So some trials and descriptive analysis.

---

Actually there was no directly coding today, but I took some additional algorithm and data structures courses under Kodluyoruz / Patika Data Science Bootcamp homeworks.

---

Today it was first day of my "Enerjisa Data Science Bootcamp" with Kodluyoruz. We talked mostly general things. Actually there was no any coding.

After course I solved some interview algorithm problems.

Rest of the day I rerun some of the github repos about CNNs.

---

---

In my data engineering bootcamp we talked about spark, etl and elt and their use cases about it.

Actually today was not enough efficient for me.

---

---

I studied ANNs on my old regression tasks. Actually I know and I belive that for tabular tasks ANNs are not good. Mostly tree based models kick ANNs ass but I tried different combinations to improve flexibility of my knowledge on deep learning.

---

Today I added the x-days-of-process heatmap (like github commit heatmap) to this page.

And I'll take GoLang course rest of the day.

---

Today after work I concated Flask blog website and my dash app [this one](https://github.com/bilative/weekend-end-to-end-ds-project). I give a "a tag" somewhere and it direct to dash dashboard page with same base url. So there is only 1 app, I didnt combined 2 different app, I concated them into 1 file. This was good experience for me, bc I've been thinking about this for a long time if this is possible or not.

---

Along with these 2 days in day time I worked about flask to create a blog website for myself. I wanna host my own blog website, with 4 main pages and subpages for 2 concepts. Acutally my process gone well. Firstly I built copy of this website (github pages). Github pages is static website, and I think it is not enough usefull. Bu I build copy of this with Flask, and I made it dynamic. I related my daily posts with databse and Flask backend, It's look is good now. There is some additional features. MAybe in short term I can carry my page to my own domain name.

In night times I took course about GoLang, actually I think this 12 days with GoLang could be good for me. Maybe I can create a repo about GoLang, and improve myself in long term. Why Not??

---

Hey, I've been accepted to a data science bootcamp. I will be in a 6-week bootcamp, It is starting next week.

And today after work I took a 3 hours basic GoLang course. Tomorrow I'll start a GoLang bootcamp at the same time, this study class will take 12 days, I wanna be ready enough for this journey. And yeap, Go looks cool.

---

---

I took care of my non-coding work.

---

After work I decided to finish coding part of my final chapter of dataviz blog series. And Yeap, I finished it, except finilizing color tones of the cards :D This is a really hard task you know, I cant decide. I can share the steps I took when I create a dashboard with python:

1. Locate cards on layout for plots,
2. Fill the cars of plots by basic plots with target types,
3. Add buttons and dropdown etc,
4. Rebuilt plots with the real data,
5. Relate and activate buttons/ dropdowns with plots,
6. Make interface prettier and make codes cleaner,

You can check the latest version:

---

Today I there was a lesson of my Data Engineering Bootcamp. I took some knowledge about ETL, ELT, Datalake solutions by scenarios.

And I solved 5-6 questions from leetcode. Robber, and buy-sell stock question was kinda good, you can check them here.

---

I upped the gear in my deep learning process. During these days, I read the second part of Geron's book "Hands on ML". Actually I read the first part of older version of the book last year, and I liked it so much. But first edition of the book was about tensorflow 1, so I read early release of second edition with tensorflow 2.

I decided to specialize in tensorflow for 3-4 months, if necessary I will adapt to pytorch later.

At the same time I applied and observed the steps on code by github repo.

And I upgraded my local tensorflow to GPU version. Actually after all things I realized that it was so easy to configure CUDA and CuDNN but they took so much time.

---

---

I started to study on deep learning again. Actually I had notes from last year about CNNs, I checked them again and I took some courses. And I'm in a 4 days mini bootcamp(with a community) about deep learning. I checked some examples and apply them step by step to understand to idea behind them. Actually I really refreshed my knowledge again and my passion increased again.

I decided to take photos of 5-6 different items more than 100 times every of them and crete my own image dataset. And I'll upload it to kaggle and analyze them with CNN. This is good idea I think, we'll see....

---

I focused to tensorflow today. I followed some documantations and some github repos. Actually I found myself little weak about the preprocessing steps of image processing CC applications. But I made good progress today.

---

Today I did some practive with google cloud services. Actually I didnt use them deeply, only the thing I use are basics. But I focused to use them from terminal. The tools I practiced were:

Pub / Sub

Big Query

Cassandra (from marketplace)

And Data Store

---

---

I updated my twitter-earthquake-bot couple days ago. And I faced with some problems. These are:

Kandilli Rasathanesi could update earthquake infos after couple minutes. And the update info breaks my web scraping script. So I solved this step with additional "try-except rows".

And I added image of the map that earthquake occured. But this required some searhing process, bc twitter API want to read images from local. So I should did convert map images from html to png and save. Views of maps in python geoplot libs are in html form, this took some time from me.

And I specified a threshold for earthquake so my bot alerts for earthquakes that greater than 4.

(I added the photo afer 20 days bc there was no clear output bc of the threshold last 20 days)

---

12 Mostly Only thing that I do is taking Java course for last 10 days. Actually I remember that there was maybe 2 days taht I do nothing. Actually I like my process, I learned so much about the idea of basic concepts of the Java programming.

Actually I'm feeling like I should carry some of my tasks Python to Java.

I started to solve some Hackerrank problems to make my Java skills to ready for different problems.

I started to read Clean Code I know that I'm not writing my codes clean on neither Python nor Java, so I should sharp my weapons.

I'm taking notes well.

There is some day offs days in this time interval!

---

Today I hacked. Actually it's not purely hacking :d Data in the cloud mongodb database stolen. But actually there was only scraped data about second hand cars. So no big deal, I increased security level of my db.
Actually I'm kinda lucky bc My database has blown up in couple of weeks (so quickly). And on my baby steps I didnt any big project so there was no any important data. If they didnt hacked me today, Maybe later I could made big projects with no enough security. I'm feeling lucky (and dumpy :d)

---

I wrote 4th chapter of my blog on sisterslab. Actually there is no so much thing but I created a dashboard for this. You can find it here.

---

I'm taking Java course nowadays. My aim is not only taking some courses, I wanna start to use Java in my projects.
Till now I learned syntax and some basics, and now I'm trying to be better on OOP with java. The process is going very good.

---

Actually today I did almost nothing. But I built my portfolio github-page and my xdaysofcode page (So current page :D).

---

---

During these 2 days I made a end-to-end project. Project and all the details are here.

---

---

Today I deployed my twitter-eartquake-bot application on cloud linux machine. Normally I'm using linux with interface but on cloud machine there is no interface, so deploying steps required some new codes to write for me. I learned usefull things today like:

ssh connection to remote machine with windows command prompt (ssh my_user@my_ip)

docker save and load image command to send saved image to remote machine

scp command to send saved image to remote machine (scp windows_path/file.tar linux_user@linux_ip:/path)

And at the same time I used cli connection to my docker container.

I used to do these with interface mostly, So today is so today has been a lot of fun for me. And twitter bot is active now 7/24, if there is a earthquake bigger than 4, bot will tweet it by my twitter account.

---

Today I rent a cloud machine (linux) for 12 months. I had no so much experience with linux, I know some basics but I wanna do more. Today:

I raised my machine,

Created new user except root,

• Installed ubuntu GUI
• And I removed it after couple hours. Bc there was a problem with speed, it was not clear to use (center of my machine is Germany, distance could be problem)

Installed python and docker and tested it.

---

Today I searched some Java apps. And I solved some entry level Java problems on Hackerrank. I'm at the begining of the Java. But I'm taking small steps. In short term maybe I wont be good at it but, in long term (maybe in 2 year) I know that I'll use it fluently.

---

---

Today I took a google cloud certification course.

---

I study about REST Api more today.

• This one

---

Today me and one of my friend checked my projects (smth like code review). And we decide that I can make my codes more efective.

• I redesigned some of my projects design to fit "Model Viev Controler".
• I eliminated repeated functions or lines. (for example)

We decided to do code reviews more. In this way I'll improve my skills by perspective of other experts, and avoid from mistakes.

---

Today I started to analyze the data I scraped online store. But after start to know data and analyze I realize that variation on independent variables are not enough. I was aim to predict Prices' of the products but prices are seem not continious variable to use regression. Im talking like this bc prizes are like 6499, 6999, 7499...

• There are about 3600 product but there is 18 different price labels.
• So this is smth like classification problem, and I dont find smth true to use regression or classification in this situation.

---

If I wanna have data engineering skills, I think I should improve my level of knowledge in REST API. And actually I'm having fun while handling with html procedures.

I followed a path like:

• I took some steps about how can I create a token in my APIs, and How can I verified users.
• I used Flask for APIs, and JWT.

---

After working day, I didn't do something important today.

I followed a path like:

• I attended to lesson of my course and we examined use cases of cassandra and some other NoSQL dbs.
• I solved little details around my data that I web scraped from online sales website. I'll start to analyze in couple days.

---

Today I study on MongoDB in the course I take. Actually I'm using mongoDB in office everyday but I'm mostly using it on my apps to store images. In the course mostly we study on tabular data. It was good, and enjoyful.
And today I solve some problem in an exam. There was an sql question in the exam, and it was smth like this:

There are 2 table as above. And show us the people who talked on the phone less than or equal to 15 minutes.

• This is good because I didnt face with a problem required 1 vs 2 Inner Join like this. And I didnt know that this is possible.
• But my solution was correct, it was smth like this:

SELECT TABLE2.name FROM TABLE2
INNER JOIN TABLE1 ON TABLE2.phone_number = TABLE1.caller OR TABLE2.phone_number = TABLE1.callee
GROUP BY TABLE2.name
HAVING sum(calls.total_time) <= 15
ORDER BY TABLE2.name

---

I solved some hackerrank problems and I read "Yapay Zeka Uygulamalari (4th edition)" from Prof. Dr. Cetin Elmas. Actually I bought thos book couple weeks ago, and today I have started to read it and apply. I refreshed my knowledges and I started to learn some mores.

---

I started to handle with the data I collected yesterday. None values and the data with no standard is a big problem. Today I worked on them about 2 hours. But there is more to do. None and nan are not acting same. I understood this so bad.

Making metrics standard

• cm, mm, or meters,
• screen resolutions 27'', 27' , 27 inch or 27INCH,
• freedos, dos or FreeDOS

I took 5k+ rows of data and more than 150 features.

sample code

---

In last days I spent my time to find a topic and a good website to do web scraping. And today i collected some of them from a online sales website. I wanna make a end to end data science project, so this one is first step. I collected data about phones, laptops, all in one pcs' and tablets. Now I have a real world dataset with more than 5k rows. Now there is so much things to do like cleaning/ standardizing.

I followed a path like:

• Taking urls of whole pages
• Taking urls of whole products on each pages
• Taking features and values of all products

I took 5k+ rows of data and more than 150 features.

sample code