Skip to main content

Posts

spark-scala-python

Recent posts

spark-Pyspark

 spark : in-memery computation spark : Data extracting from hard disk and store into ram. and each step operation are store in ram and only after completion of job it write to hard disk. hadoop mapreduce : it perform some operation and store to hard drive every time for each step. it read and write from hard disk every step of the job. so latency is high. Lazy execution : when applying function to read the data it does not read because we do not performing any opearaiton. it does not read data until we perform some operation or computation. suppose we do in pandas pd.read_csv it read data store in ram. Parallel Processing : distributed the data into different cluster and stored in nodes. batch processing and real-time processing ex. credit card transaction. genuine or fake

Logistic Regression

 Logistic Regression : logistic regression is a supervised learning algorithm used for classification problems. It is mainly used for binary classification. the target variable is categorical. the algorithm predicts the group which the current object belong.  it predicts the group by estimating the probabilities based on weightage relationship between dependent and independent variable.  for this probabilities it uses the logistics function. the task of logistic function is to take any real value and map it between 0 and 1. then this values are converted into binary values 0 and 1 using threshold value. default threshold value is 0.5

Bagging and Boosting

  What is an Ensemble Method? The ensemble is a method used in the machine learning algorithm. In this method, multiple models or ‘weak learners’ are trained to rectify the same problem and integrated to gain desired results. Weak models combined rightly give accurate models. Bagging Bagging is an acronym for ‘Bootstrap Aggregation’ and is used to decrease the variance in the prediction model. Bagging is a parallel method that fits different, considered learners independently from each other, making it possible to train them simultaneously. Bagging generates additional data for training from the dataset. This is achieved by random sampling with replacement from the original dataset. Sampling with replacement may repeat some observations in each new training data set. Every element in Bagging is equally probable for appearing in a new dataset.  These multi datasets are used to train multiple models in parallel. The average of all the predictions from different ensemble models i...

Windows 11 shortcut

 # to set the windows 10 right click context menu reg add "HKCU\Software\Classes\CLSID\{86ca1aa0-34aa-4e8b-a509-50c905bae2a2}\InprocServer32" / f paste this  command in cmd # go back to windows 11 context menu reg delete "HKCU\Software\Classes\CLSID\{86ca1aa0-34aa-4e8b-a509-50c905bae2a2}" / f paste this command in cmd

cammand for installing library in python

 Command for installing in jupyter notebook:                pip install library_name                ex. pip install nump installing from anaconda prompt:           1. pip install numpy           2.   conda install -c conda-forge matplotlib search for conda command for matplotlib and go to official website. Installing from anaconda navigator easy. Somtime give error then open as administrator

Tabel of content in Jupyter notebook

  SourceForge uses markdown syntax everywhere to allow you to create rich text markup, and extends markdown in several ways to allow for quick linking to other artifacts in your project. Markdown was created to be easy to read, easy to write, and still readable in plain text format. Links Reference Links Artifact Links User Mentions Basic Text Formatting Blockquotes Preformatted Text Lists Tables Headers Horizontal Rules Images Videos Escapes and HTML More Headers Table of Contents Code Highlighting Includes Neighborhood Notifications Project Info Macros Thanks Links Most URLs will automatically be turned into links. To be explicit, just write it like this: <http://someurl> <somebbob@example.com> Output: http://someurl somebbob@example.com To use text for the link, write it: [like this](http://someurl) Output: like this You can add a *title* (which shows up under the cursor): [like this](http://someurl "this title shows up when you hover") Output: like this Refe...