Mikrotik in real life,full scale and low budget ISP.pdf

Mikrotik in real life, full scale and low budget ISP [email protected]

additional presentation

How do the youngest country in the world ISPs run their bussiness more efficient, and more reliable with Mikrotik

about me • Working in ISP industries since 1994

• Currently working as consultant engineer for asia pacific oceania countries company & organization

Definition • Real life • Not in simulation, or lab scale • In bussiness, operational, still operational

• Full Scale • Have ASN, buy transit, peering • Connect to IX

• Low Budget • Is low budget

Disclaimer • This presentation will not talk in depth about BGP, OSPF & Traffic Engineering • I just share simple example, and how to do it with Mikrotik • It is real case, some IP/AS is fake, for security

Before • Cisco 7200 VXR    

Border Router BGP Peering to Transit Provider BGP Peering to Local IXP Customer Access Router

• IBM e Series

 FreeBSD / Quagga  BGP Peering to Local IXP  BGP Peering to IP Transit Customer

• Problem

 Expensive Router  Difficult to Maintain

Reason to Upgrade • • • • •

Efficiency Performance Maintenance Cost Growth

After • CCR 1036-8 – Transit

 OSPF  BGP Peering to Transit Provider

• CCR 1036-8 – IXP  OSPF  BGP Peering to IXP

• CCR 1036-12 – Core  OSPF  BGP Route collector

• CCR 1009 – Access  Static Routing, VLAN, Trunk  Management Router

• RB750 – OOB  VPN

Physical Network Diagram

Configuration Pre-config  Turn off unused service features • Web,telnet,ftp,etc  Winbox / SSH only available from Remote Access IP  Change default port

Configuration Turn off unused packages features  Disable features/packages

Configuration Neighbour discovery  Disable interface

 Disable MNDP on interface to IXP/Transit, some of them will handle this as a threat  Some IX/Transit require you to turn off Proxy ARP,ICMP redirects,Directed broadcast,IEEE802 Spanning Tree,Interior routing protocol broadcast,Mac layer broadcast  Read peering agreement

Configuration • Disable unused physical interface • Device name • User / Password  Proper credentials

• NTP Client  Make sure your router time is synchronized

• Latest stable OS • Disable LCD / Minimal information

Configuration OSPF between devices for IGP  for infrastructure  loopback interface, for adjacency, not only router id

Configuration iBGP between devices  TR – CR – IXP  Loopback interface peering  For carry prefixes across backbone

iBGP instance

Loopback peering

Advertise Networks Loopback interface as source

Checking

Configuration eBGP between peer / other AS  Peering  Advertise your prefixes  Filtering     

In Filter -> how we send the traffic Out filter -> how they will send the traffic Standard regexp Use template for filter Organize filter using jump

 Traffic engineering, routing policy, follow BGP BCP

Peering  Use your AS, peering IP, peer AS  Prepare your in/out filter

Advertise your prefix

 Announce your aggregate from registry

 Use blackhole type route for pull-up route  Put on core router, not border

Announce your aggregate, for internet stability

Routing Filters

 In Filter -> how we send the traffic -> our routing table  Out filter -> how they will send the traffic -> their route to our AS  Template  In Filter  Discard prefix from other peering AS  Accept prefix from peering AS  Discard our own prefixes  Discard RFC5735 prefixes  Discard prefix longer than 24  Out Filter  Allow only our prefixes to be announce  Use jump for organize your rule

 Regexp    

. - any single character ^ - start of the as-path $ - end of the as-path _ - matches comma, space, start and end of as-path

Traffic Engineering, Policy Routing  BGP Attribute 

http://wiki.mikrotik.com/wiki/Manual:BGP_Best_Path_Selection_Algorithm

 Routing scenario, multihomed

 Redudancy  Load sharing  Local traffic goes to and from local peer

References • • • •

NANOG / APNIC BGP Tutorial BGP Filtering with Router OS – 2013 MUM Croatia by W. Maia Routing Security – 2011 MUM Hungary by W. Maia BGP and OSPF Implementations – 2011 MUM Hungary by D. Burgess

Configuration Access Router  Plain Static Routing for customer  Bandwidht Manager  Controlled by Management Server

Remote Access Server (RB750)  Secure VPN PPTP/L2TP  OOB - Connection from other ISP

Configuration Bandwidht Manager  Strategy  Mark packet came from AC router for Upload  Mark packet came from TR/IX router for Download  Done at Core Router

 International / Local Simple Queue / Queue Tree  You can use transparent traffic limiter  http://wiki.mikrotik.com/wiki/TransparentTrafficShaper

Configuration

 Bridge / Routing Configuration  International / Local Management  Routing List, ref : http://mikrotik.co.id/artikel_lihat.php?id=23  Custom Scripting -> export routing from bgp router

Configuration

Transparent  Create Bridge Interface  Marking, check packet flow diagram

Configuration

Management Server    

Don’t touch my router Simple Mikrotik ROS API Call Automatic IP / VLAN / BW Allocation Automatic client activation / cut-off

Screenshot Transit Router

IX Router

Maintenance ROS upgrade strategy

 Use stable/current only

 RouterOS current release 6.XX  RouterOS bugfix release 6.XX.Y

 Read Changelog  Upgrade wisely

 Improving system stability

Config backup

 Simple script

Documentation  

Everything Log / Syslog ex: syslog-ng

additional presentation

How do the youngest country in the world ISPs run their bussiness more efficient, and more reliable with Mikrotik

History of iNet Timor 

1999 – Referendum for Freedom

 

2000 – Telstra start cellular telephone 2003 – Timor Telecom : Voice (GSM/PSTN) Telstra iNet : Data Internet (ADSL/Dialup/Wireless)

Before Mikrotik Network scale      

30Mbps Upstream One main hub Dialup / ADSL Services 3 Wireless BTS around Dili VSAT Backbone 20 Client

Using well known product Cisco Router Cisco Switch Nortel/Paradyne DSLAM Avaya / Cisco / Breezecom  Cisco 800 / 2500 CP Router  Airlive CPE    

Past

Problem • Power line quality are bad, devices easy to damage • Time to deliver replacement devices • From HQ (2 weeks) • From order to deliver, 15 day minimum

• High down time • Expensive, almost impossible to have spare • High cost CPE

Mikrotik • •

• •

•

2006 – 2007  RB230, RB132, RB133, RB532 (RouterOS v2) as Wireless Infrastructure 2008  RB1000 (RouterOS v3), as experimental access router 2009  RB750 as CPE router, replace Cisco 700,800,2500 2010  RB1100 as Edge Router – Cisco replacement  BGP/OSPF (One default route only, One Full Routing Table)  RB1100 as Bandwidht Manager  HTB, good but complicated  Simple Filter Rule  RB1100 – As Distribution Router  Plain static routing 2013  Next step with CCR1036  Using Simple Queue as RouterOS 6, lot easier than HTB and faster

After Mikrotik Network scale  150Mbps Upstream  VSAT & Fiber Multihoming

 Wireless & GPON Services  16 Wireless BTS around Dili, 5 remote BTS  > 400 Client

Expansion • 3G/4G LTE Access Point (Canceled 2013) • GPON FTTH, Mikrotik ONT/ONU only • Solar Powered remote BTS    

100W Solar panel + Battery GSM remote switch RB750UP / hEX POE Lite for controller RB433, RB911, Metal

The Dude

CCR1036 400 client simple queues ? No Problem, we did that

Result • Reducing Expenses  Reducing capex  Reducing customer cost

• Fast to deliver, Fast to replace, Low down time  We have cold spare devices  If we dont, we can get it less than 12 hours

• Easy to operate and maintain  Winbox easy to use

• Open lot of possibilities,  Exploit all of technology available on ROS

Last Update iNet start using O3b     

Medium orbit Satellite, not geo stationer 360 minutes contact per satellite 300ms latency, not 500ms anymore Required two autotracking dish Using Mikrotik to do VRRP

Last.. • Don’t be affraid to use RouterOS on your ISP • Don’t be embarassed if you already use ROS • Router OS have complete features for ISP

Thank You