Recent Entries 10
- pattern minor 112d agoHTTP redirects after login, registration, and logoutI am using following classes/interface to redirect user (after login, register, logout etc.) File: RedirectInterface.php ``` interface RedirectInterface { public function getUrl($customerId = null); } ``` File: LoginRedirect.php ``` class LoginRedirect implements RedirectInterface { public function getUrl($customerId = null) { // do some business logic here to get url $url = '/account/some-customer'; return $url; } } ``` File: RegisterRedirect.php ``` class RegisterRedirect implements RedirectInterface { public function getUrl($customerId = null) { // do some business logic here to get url $url = '/welcome/some-customer'; return $url; } } ``` File: RedirectFactory - Creational design pattern (static factory) ``` class RedirectFactory { public static function build($type, $customerId) { if ($type == 'login') { return new LoginRedirect($customerId); } else if ($type == 'register') { return new RegisterRedirect($customerId); } throw new InvalidArgumentException ('Invalid redirect type.'); } } ``` Usage: ``` // 1. Usage: Somewhere in customer registration code $redirectUrl = RedirectFactory::build('register', 102)->getUrl(); header('Location: ' . $redirectUrl); // 2. Usage: Somewhere in customer login code $redirectUrl = RedirectFactory::build('login', 102)->getUrl(); header('Location: ' . $redirectUrl); ``` If you were given a chance to refactor this code. How would you have re-developed it?
- pattern minor 112d agoResumable download with System.net.HTTPClientThis is a Delphi class, based on `System.net.HTTPClient` with a function for downloading a file from a URL and saving on a filename destination: ``` function Download(const ASrcUrl : string; const ADestFileName : string): Boolean; ``` The main feature is the ability to suspend or resume partial download. ``` unit AcHTTPClient; interface uses System.Net.URLClient, System.net.HTTPClient; type TAcHTTPProgress = procedure(const Sender: TObject; AStartPosition : Int64; AEndPosition: Int64; AContentLength: Int64; AReadCount: Int64; ATimeStart : Int64; ATime : Int64; var Abort: Boolean) of object; TAcHTTPClient = class private FOnProgress: TAcHTTPProgress; FHTTPClient: THTTPClient; FTimeStart: cardinal; FCancelDownload: boolean; FStartPosition: Int64; FEndPosition: Int64; FContentLength: Int64; private procedure SetProxySettings(AProxySettings: TProxySettings); function GetProxySettings : TProxySettings; procedure OnReceiveDataEvent(const Sender: TObject; AContentLength: Int64; AReadCount: Int64; var Abort: Boolean); public constructor Create; destructor Destroy; override; property ProxySettings : TProxySettings read FProxySettings write SetProxySettings; property OnProgress : TAcHTTPProgress read FOnProgress write FOnProgress; property CancelDownload : boolean read FCancelDownload write FCancelDownload; function Download(const ASrcUrl : string; const ADestFileName : string): Boolean; end; implementation uses System.Classes, System.SysUtils, Winapi.Windows; constructor TAcHTTPClient.Create; // ----------------------------------------------------------------------------- // Constructor begin inherited Create; // create an THTTPClient FHTTPClient := THTTPClient.Create; FHTTPClient.OnReceiveData := OnReceiveDataEvent; // setting the timeouts FHTTPClient.ConnectionTimeout := 5000; FHTTPClient.Respo
- pattern minor 112d agoTokenizing texts from Gutenberg archive for analysisI am writing a program to analyze books from the Gutenberg archive. The program takes the title and URL and finds the text and downloads it. Then it goes through the text and tokenizes it. Here is the code: ``` from urllib import request import nltk import os.path canon_titles = [ 'Moby Dick', 'Great Expectations' ] canon_urls = [ 'http://www.gutenberg.org/files/2701/2701-0.txt', # Moby Dick 'http://www.gutenberg.org/files/1400/1400-0.txt' ] # Canon_beginnings is the location of where the book actually begins in # the text file (skips headers, publisher info, etc.) canon_beginnings = [ 28876, 886 ] # canon endings exists just to grab a small amount of text for prototyping canon_endings = [x + 500 for x in canon_beginnings] canon_raw = [None] * len(canon_titles) canon_tokens = [None] * len(canon_titles) canon_words = [None] * len(canon_titles) canon_words2tokens = [None] * len(canon_titles) canon_pos = [None] * len(canon_titles) # Now I combine all these together into a dictionary canon_dict = {z[0]: list(z[1:]) for z in zip(canon_titles, canon_urls, canon_beginnings, canon_endings, canon_raw, canon_tokens, canon_words, canon_words2tokens, canon_pos)} # Now I go through each title in the dict and see if I already have the text (I rerun this in Jupyter Notebook sometimes) # And if not I grab it from online for x in canon_dict: print("Working on {}".format(x)) if canon_dict[x][3] == None: print("{} does not already exist, grabbing the text".format(x)) url = canon_dict[x][0] response = request.urlopen(url) canon_text_draft = response.read().decode('utf8') canon_dict[x][3] = canon_text_draft[canon_dict[x][1]:canon_dict[x][2]] else: print("Already have this text, skipping") # OK, now we'll tokenize, do parts of speech, etc. def tokinze_text(raw_text): tokens = nltk.word_tokenize(raw_text) return tokens # Now let's find the tokens
- pattern minor 112d agoMaking multiple http request Angular2I need to get some ids from contacts and contactgroups to be able to collect the data. My example works but i dont know if this is the way to go. ``` getIdArray() { let contactIds = []; let groupIds = []; let contacts = this.http.get('Contacts/').map(res => res.json()); let contactGroup = this.http.get('ContactGroups/').map(res =>res.json()); Observable.forkJoin([contacts, contactGroup]).subscribe(res => { for (let b of res[0].contacts) { console.log(b.contactId); contactIds.push(b.contactId) } for (let b of res[1].contactgroups) { console.log(b.contactgroupId) groupIds.push(b.contactgroupId) } this.loadContacts(contactIds); this.loadGroups(groupIds); }); } loadContacts(contactId) { console.log("ids" + JSON.stringify(contactId)); Observable.forkJoin( contactId.map( i => this.http.get('Contacts/' + i) .map(res => res.json()) ) ).subscribe(data => { for (let b of data) { console.log(b); } }); } loadGroups(groupIds){ console.log("ids" + JSON.stringify(groupIds)); Observable.forkJoin( groupIds.map( i => this.http.get('ContactGroups/' + i) .map(res => res.json()) ) ).subscribe(data => { for (let b of data) { console.log(b); } }); } ```
- pattern minor 112d agoLink checker using Go channelsI've started to learn Golang and channels in it. I decided to write simple application - recursive link checker. Given some URL it tries to retrieve pages, parses them and goes deeper. Here's a code of first version. Some questions: - I use counter `urlsInProcess` to understand when all tasks are done. But it looks a little bit awkward, isn't it? - I launch manually several parsers and fetchers. Should I use here WaitGroups? - Not sure about error treatment. How to do it better? - How effective is variable transmitting between goroutines? Are there some unnecessary copying? - Has it some race conditions? For example can program got message from channel `chanTasksFinished` before than from channel `chanTasksToFetch`? In this case we will exit before all tasks are done. ``` package main import ( "fmt" "golang.org/x/net/html" "io/ioutil" "log" "net/http" "strings" ) type url string type TaskStatus int const ( _ TaskStatus = iota TaskStatusNew TaskStatusToParse ) type Task struct { url depth int resp *http.Response //body *[]byte } var chanTasksToFetch = make(chan Task) var chanTasksToParse = make(chan Task) var chanFetchersIn = make(chan Task) var chanTasksFinished = make(chan Task) var mainWait = make(chan interface{}) var data = make(map[url]Task) func getHref(t html.Token) (ok bool, href string) { // iterate over all of the token's attribs for _, a := range t.Attr { if a.Key == "href" { return true, a.Val } } return } func extractLinks(page string) (urls []url) { // extracts links from page and sends them into the url_channel tokenizer := html.NewTokenizer(strings.NewReader(page)) for { token_type := tokenizer.Next() switch { case token_type == html.ErrorToken: log.Println(fmt.Sprintf("Error token_type: %s. Error: %s", token_type, tokenizer.Err())) return case token_type == html.St
- pattern minor 112d agoSimple Rust WebserverI started learning Rust today and for my first project created a simple webserver. I don't have a clear vision about Rust programming and I tried to write it with my previous knowledge in Python and PHP. ``` use std::io::{Read, Write,BufReader, BufRead}; use std::net::{TcpListener, TcpStream}; use std::fs::File; use std::path::Path; fn main(){ loop{ let listener = TcpListener::bind("localhost:8000").unwrap(); let stream = listener.accept().unwrap().0; read_request(stream); } } fn read_request(stream: TcpStream){ let mut lines = String::new(); let mut reader = BufReader::new(stream); reader.read_line(&mut lines); let mut vec_line = lines.split_whitespace(); let mut requested_page = vec_line.nth(1).unwrap().to_string(); requested_page = requested_page.replace("/",""); if requested_page == "" { requested_page = String::from("index.html"); } let mut response = String::new(); let path = Path::new(&requested_page); println!("{}",requested_page); let mut status = 200; if !path.exists(){ response = String::from("Not Found!"); status = 404; } else { let mut file = File::open(&requested_page).expect("Unable to open file"); file.read_to_string(&mut response); } send_response(reader.into_inner(), &response.to_string(), status); } fn send_response(mut stream: TcpStream, res :&str, status :u32){ let response = format!("{}{}{}{}","HTTP/1.1 ",status," OK\n\n",res); stream.write_all(response.as_bytes()).unwrap(); } ```
- pattern minor 112d agoParsing HTTP Headers in C++I am building a web server from scratch and trying to make certain tasks faster by embedding C code into the mix for performance. Specifically I'm worried about how the `std::string` class with the `.find()` and other functions compare to straight pointer arithmetic. ``` #include #include #include std::map http_request; void parse_header( void * ); int main() { char * msg= "GET / HTTP/1.1\r\n" "Host: 192.241.213.46:6880\r\n" "Upgrade-Insecure-Requests: 1\r\n" "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n" "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/602.4.8 (KHTML, like Gecko) Version/10.0.3 Safari/602.4.8\r\n" "Accept-Language: en-us\r\n" "Accept-Encoding: gzip, deflate\r\n" "Connection: keep-alive\r\n\r\n"; parse_header( msg ); } void parse_header( void *msg ) { char *head = (char *) msg; char *mid; char *tail = head; if( sizeof( msg ) == 0 ) { return; } // Find request type while( *head++ != ' '); http_request[ "Type" ] = std::string( ( char * ) msg ).substr( 0 , ( head - 1) - tail ); // Find path tail = head; while( *head++ != ' '); http_request[ "Path" ] = std::string( ( char * ) msg ).substr( tail - ( char *)msg , ( head - 1) - tail ); // Find HTTP version tail = head; while( *head++ != '\r'); http_request[ "Version" ] = std::string( ( char * ) msg ).substr( tail - ( char *)msg , ( head - 1) - tail ); // Map all headers from a key to a value while( true ) { tail = head + 1; while( *head++ != '\r' ); mid = strstr( tail, ":" ); // Look for the failed strstr if( tail > mid ) break; http_request[ std::string( ( char * ) msg ).substr( tail - ( char *)msg , ( mid ) - tail ) ] = std::string( ( char * ) msg ).substr( mid +
- debug minor 112d agoElegantly handle github API requests exceptionsI have this code which creates a team using the github API and returns the team-id(success) or -1(failure). My problem is with the error handling. When the request responses with an error status code I try to deal with a few different types of failure. It looks like a mess! Is there any way to make it more readable/elegant? I'd also like to know if I am catching all the reasonable exceptions that can be thrown by requests. ``` def create_team(org,team_name): ''' Try to create a team and return its id if already exisits return id ''' try: data = { "name": team_name, "permission": "push" } headers = {'Authorization': 'token '+OAUTH_TOKEN} response = requests.post(GH_API_URL+'orgs/'+org['login']+'/teams', headers=headers, data = json.dumps(data)) response.raise_for_status() #throw exception if request does not retun 2xx #http status is 2xx, team must have been created json_response = json.loads(response.text) return json_response['id'] #get id of new team from response except requests.exceptions.HTTPError as e: if response.status_code == 422: #Unprocessable Entity json_response = json.loads(response.text) if json_response['errors'][0]['code'] == 'already_exists': #Unprocessable because already exists print(' not created, team already exists!!!') #get the id of existing team team_id = get_teamID_from_name(org, team_name) return team_id else: #Unprocessable for some other reason print(' HTTP error: '+str(e)) else: #other HTTP error != 422 print(' HTTP error: '+str(e)) except requests.exceptions.RequestException as e: print('Connection error: '+str(e)) return -1 ``` In case you were wondering why I don't just check
- pattern minor 112d agoRESTful HTTP Post with falconI wrote a small aggregator app that aggregates values from a json http post request and outputs the aggregated values. Now the aggregator function is somewhat large but the output appears correct. I start it with gunicorn: ``` $ gunicorn --workers 2 --threads 4 aggregator:api ``` I test it with curl : ``` $ curl -X POST -d @aggregator.json http://localhost:8000/aggregate ``` It outputs the aggregated values. ``` import falcon import json from datetime import datetime from dateutil import tz import requests import time from_zone = tz.gettz('UTC') to_zone = tz.gettz('Europe/Stockholm') class DataService: def on_post(self, req, resp): data = json.loads(req.stream.read().decode('utf-8')) # output the data, we could write it to persistent storage here print(data) class AggregatorService: def zero_if_none(self, value): if value is not None: x = int(value) else: x = 0 return x # return the start day of consumption e.g. 1 for 2014-12-01 def get_day_start(self, hours): return int(datetime.fromtimestamp( int(hours[0][0]) ).strftime('%d')) # return the month number e.g. 2 for February def get_month_start(self, hours): return int(datetime.fromtimestamp( int(hours[0][0]) ).strftime('%m')) # return the day of month number e.g. 5 for 2015-01-05 def get_day(self, hour): return int(datetime.fromtimestamp( int(hour[0]) ).strftime('%d')) # return the month number for a timestamp def get_month(self, hour): return int(datetime.fromtimestamp( int(hour[0]) ).strftime('%m')) def on_post(self, req, resp): data = json.loads(req.stream.read().decode('utf-8')) hours = data['hours'] day_start = self.get_day_start(hours) month_start = self.get_month_start(hours) aggr_daily_wh = 0 aggr_mon
- pattern minor 112d agoScraping GitHub for security vulnerabilitiesI haven't done much Python programming, but I really like the language so I've been using it for side projects that I work on. The problem with this is that I don't have the opportunity to have my peers review my code and give suggestions. I've been working on a project that will be scraping GitHub looking for security vulnerabilities. I've created a separate file in the project that contains all functions that interact with GitHub's API. ``` import requests import re import base64 import os def getRepos(since=0): url = 'http://api.github.com/repositories' data = """{ since: %s }""" % since response = requests.get(url, data=data) if response.status_code == 403: print "Problem making request!", response.status_code print response.headers matches = re.match(r'', response.headers['Link']) next = matches.group(0)[1:-1] return response.json(), next def getRepo(url): response = requests.get(url) return response.json() def getReadMe(url): url = url + "/readme" response = requests.get(url) return response.json() # todo: return array of all commits so we can examine each one def getRepoSHA(url): # /repos/:owner/:repo/commits commits = requests.get(url + "/commits").json() return commits[0]['sha'] def getFileContent(item): ignoreExtensions = ['jpg'] filename, extension = os.path.splitext(item['path']) if extension in ignoreExtensions: return [] content = requests.get(item['url']).json() lines = content['content'].split('\n') lines = map(base64.b64decode, lines) print 'path', item['path'] print 'lines', "".join(lines[:5]) return "".join(lines) def getRepoContents(url, sha): # /repos/:owner/:repo/git/trees/:sha?recursive=1 url = url + ('/git/trees/%s?recursive=1' % sha) # print 'url', url response = requests.get(url) return response.json() ``` The code is run from here: ``` import github import json def processRepoContents(repoContents): # for each entry in the repo for tree in repoC